Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260810 23:32:02.392875 12170 test_util.cc:268] Using random seed: -1025032594
2026-08-10T23:32:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:02Z Disabled control of system clock
I20260810 23:32:02.466706 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.190:43533
--webserver_interface=127.11.226.190
--webserver_port=0
--builtin_ntp_servers=127.11.226.148:41081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260810 23:32:02.982880 12188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:02.983867 12188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:02.984493 12188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:03.020357 12188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:03.020772 12188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:03.021114 12188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:03.021415 12188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:03.063258 12188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:41081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.190:43533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.226.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:32:03.065023 12188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:03.067150 12188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:03.097674 12198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:03.106083 12201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:03.098255 12197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:04.525224 12200 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1423 milliseconds
I20260810 23:32:04.525408 12188 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 23:32:04.526996 12188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:04.533739 12188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:04.535249 12188 hybrid_clock.cc:648] HybridClock initialized: now 1786404724535140 us; error 140 us; skew 500 ppm
I20260810 23:32:04.536470 12188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:04.550493 12188 webserver.cc:460] Webserver started at http://127.11.226.190:41729/ using document root <none> and password file <none>
I20260810 23:32:04.551829 12188 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:04.552227 12188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:04.552918 12188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:04.559204 12188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/data/instance:
uuid: "e964330cf113403a8aa5b73ddd3b31d0"
format_stamp: "Formatted at 2026-08-10 23:32:04 on dist-test-slave-kvfs"
I20260810 23:32:04.560631 12188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/wal/instance:
uuid: "e964330cf113403a8aa5b73ddd3b31d0"
format_stamp: "Formatted at 2026-08-10 23:32:04 on dist-test-slave-kvfs"
I20260810 23:32:04.573536 12188 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.002s sys 0.009s
I20260810 23:32:04.584558 12213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:04.586339 12188 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.006s
I20260810 23:32:04.586841 12188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/wal
uuid: "e964330cf113403a8aa5b73ddd3b31d0"
format_stamp: "Formatted at 2026-08-10 23:32:04 on dist-test-slave-kvfs"
I20260810 23:32:04.587291 12188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:04.641096 12188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:04.643783 12188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:04.644467 12188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:04.927790 12188 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.190:43533
I20260810 23:32:04.928098 12290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.190:43533 every 8 connection(s)
I20260810 23:32:04.931473 12188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:04.932955 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 12188
I20260810 23:32:04.933473 12170 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-0/wal/instance
I20260810 23:32:04.938290 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.189:42101
--webserver_interface=127.11.226.189
--webserver_port=0
--builtin_ntp_servers=127.11.226.148:41081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260810 23:32:04.951333 12292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:04.977543 12292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:32:05.033993 12292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } has no permanent_uuid. Determining permanent_uuid...
W20260810 23:32:05.039860 12215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
W20260810 23:32:05.053153 12292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:42101: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:05.103107 12292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } attempt: 1
W20260810 23:32:05.110682 12292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:42101: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:05.179662 12292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } attempt: 2
W20260810 23:32:05.186561 12292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:42101: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:05.278263 12292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } attempt: 3
W20260810 23:32:05.286566 12292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:42101: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:05.430294 12292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } attempt: 4
W20260810 23:32:05.436589 12292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:42101: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
W20260810 23:32:05.559453 12295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:05.560403 12295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:05.560976 12295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:05.597448 12295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:05.597890 12295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:05.598218 12295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:05.598500 12295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:05.638120 12295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:41081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.189:42101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.226.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:32:05.639840 12295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:05.641968 12295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:05.679747 12312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:05.736210 12292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } attempt: 5
W20260810 23:32:05.742830 12292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:42101: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:06.290462 12292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } attempt: 6
W20260810 23:32:06.305596 12292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:42101: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
W20260810 23:32:05.681128 12310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:05.679770 12309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:07.123409 12311 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 23:32:07.123600 12295 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 23:32:07.130065 12295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:07.135123 12295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:07.136725 12295 hybrid_clock.cc:648] HybridClock initialized: now 1786404727136665 us; error 77 us; skew 500 ppm
I20260810 23:32:07.137792 12295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:07.152297 12295 webserver.cc:460] Webserver started at http://127.11.226.189:37775/ using document root <none> and password file <none>
I20260810 23:32:07.153650 12295 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:07.153936 12295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:07.154608 12295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:07.162420 12295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data/instance:
uuid: "198f534261994047a68e1d509bb2fd8b"
format_stamp: "Formatted at 2026-08-10 23:32:07 on dist-test-slave-kvfs"
I20260810 23:32:07.163915 12295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/wal/instance:
uuid: "198f534261994047a68e1d509bb2fd8b"
format_stamp: "Formatted at 2026-08-10 23:32:07 on dist-test-slave-kvfs"
I20260810 23:32:07.175462 12295 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.004s
I20260810 23:32:07.186831 12326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:07.188604 12295 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.005s
I20260810 23:32:07.189055 12295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/wal
uuid: "198f534261994047a68e1d509bb2fd8b"
format_stamp: "Formatted at 2026-08-10 23:32:07 on dist-test-slave-kvfs"
I20260810 23:32:07.189544 12295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:07.255211 12295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:07.257799 12295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:07.258630 12295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:07.366495 12292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } attempt: 7
W20260810 23:32:07.382035 12292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:42101: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:07.495793 12295 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.189:42101
I20260810 23:32:07.495947 12405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.189:42101 every 8 connection(s)
I20260810 23:32:07.499732 12295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data/info.pb
I20260810 23:32:07.506757 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 12295
I20260810 23:32:07.507450 12170 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/wal/instance
I20260810 23:32:07.511634 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.188:45477
--webserver_interface=127.11.226.188
--webserver_port=0
--builtin_ntp_servers=127.11.226.148:41081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260810 23:32:07.513504 12406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:07.535403 12406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:32:07.584429 12406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:32:07.616914 12406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } has no permanent_uuid. Determining permanent_uuid...
W20260810 23:32:07.620936 12332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
W20260810 23:32:07.630299 12406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:45477: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
I20260810 23:32:07.680138 12406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } attempt: 1
W20260810 23:32:07.687816 12406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:45477: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
I20260810 23:32:07.756611 12406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } attempt: 2
W20260810 23:32:07.763046 12406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:45477: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
I20260810 23:32:07.854703 12406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } attempt: 3
W20260810 23:32:07.861097 12406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:45477: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
I20260810 23:32:08.004638 12406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } attempt: 4
W20260810 23:32:08.011660 12406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:45477: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
W20260810 23:32:08.065421 12410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:08.066437 12410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:08.067024 12410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:08.111861 12410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:08.112454 12410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:08.112859 12410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:08.113196 12410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:08.152752 12410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:41081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.188:45477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.226.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:32:08.154469 12410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:08.156607 12410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:08.195547 12429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:08.311609 12406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } attempt: 5
W20260810 23:32:08.318948 12406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:45477: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
I20260810 23:32:08.866689 12406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } attempt: 6
W20260810 23:32:08.888726 12406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:45477: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
I20260810 23:32:09.472801 12292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } attempt: 8
I20260810 23:32:09.511772 12292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } has no permanent_uuid. Determining permanent_uuid...
W20260810 23:32:09.530093 12292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:45477: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
I20260810 23:32:09.595970 12292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } attempt: 1
W20260810 23:32:09.602921 12292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:45477: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
W20260810 23:32:08.197310 12434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:08.202268 12410 server_base.cc:1034] running on GCE node
W20260810 23:32:08.198125 12428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:09.631711 12410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:09.636410 12410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:09.638039 12410 hybrid_clock.cc:648] HybridClock initialized: now 1786404729638015 us; error 79 us; skew 500 ppm
I20260810 23:32:09.639328 12410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:09.656489 12410 webserver.cc:460] Webserver started at http://127.11.226.188:46881/ using document root <none> and password file <none>
I20260810 23:32:09.656526 12292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } attempt: 2
I20260810 23:32:09.657915 12410 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:09.658213 12410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:09.658851 12410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
W20260810 23:32:09.662840 12292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:45477: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
I20260810 23:32:09.666698 12410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/data/instance:
uuid: "681674cb24274f10b200f264f155c2d1"
format_stamp: "Formatted at 2026-08-10 23:32:09 on dist-test-slave-kvfs"
I20260810 23:32:09.668601 12410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/wal/instance:
uuid: "681674cb24274f10b200f264f155c2d1"
format_stamp: "Formatted at 2026-08-10 23:32:09 on dist-test-slave-kvfs"
I20260810 23:32:09.682569 12410 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.006s sys 0.005s
I20260810 23:32:09.695178 12451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:09.697198 12410 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.005s
I20260810 23:32:09.697674 12410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/wal
uuid: "681674cb24274f10b200f264f155c2d1"
format_stamp: "Formatted at 2026-08-10 23:32:09 on dist-test-slave-kvfs"
I20260810 23:32:09.698161 12410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:09.739496 12292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } attempt: 3
W20260810 23:32:09.748238 12292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:45477: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
I20260810 23:32:09.765293 12410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:09.768152 12410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:09.769027 12410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:09.903829 12292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } attempt: 4
W20260810 23:32:09.912967 12292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:45477: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
I20260810 23:32:09.949862 12406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } attempt: 7
W20260810 23:32:09.972105 12406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:45477: Network error: Client connection negotiation failed: client connection to 127.11.226.188:45477: connect: Connection refused (error 111)
I20260810 23:32:10.085629 12410 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.188:45477
I20260810 23:32:10.085916 12542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.188:45477 every 8 connection(s)
I20260810 23:32:10.089797 12410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/data/info.pb
I20260810 23:32:10.091881 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 12410
I20260810 23:32:10.092576 12170 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-2/wal/instance
I20260810 23:32:10.113281 12544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:10.151211 12544 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:32:10.209606 12292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } attempt: 5
I20260810 23:32:10.227241 12544 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:32:10.257804 12544 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:32:10.270025 12292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Bootstrap starting.
I20260810 23:32:10.286798 12292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:10.296139 12292 log.cc:824] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:10.308482 12544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1: Bootstrap starting.
I20260810 23:32:10.311673 12292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: No bootstrap required, opened a new log
I20260810 23:32:10.329200 12544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:10.337307 12544 log.cc:824] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:10.355809 12544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1: No bootstrap required, opened a new log
I20260810 23:32:10.384641 12292 raft_consensus.cc:348] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:10.386471 12292 raft_consensus.cc:374] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:10.387224 12292 raft_consensus.cc:729] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e964330cf113403a8aa5b73ddd3b31d0, State: Initialized, Role: FOLLOWER
I20260810 23:32:10.389106 12292 consensus_queue.cc:260] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:10.399839 12559 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:10.401844 12559 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:10.409746 12292 sys_catalog.cc:564] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:32:10.417824 12221 tablet.cc:2369] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:10.430385 12544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:10.432691 12544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:10.433501 12544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681674cb24274f10b200f264f155c2d1, State: Initialized, Role: FOLLOWER
I20260810 23:32:10.435773 12544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:10.447860 12564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:10.450178 12564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:10.452077 12544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:32:10.535506 12458 tablet.cc:2369] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:32:10.541056 12582 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:32:10.551627 12582 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:32:10.596129 12564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:10.596987 12564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:10.509618 12188 master_runner.cc:186] Error getting master registration for 127.11.226.189:42101: OK, instance_id { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" instance_seqno: 1786404727425148 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:32:10.609609 12188 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:32:10.615725 12564 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:10.620464 12559 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:10.620338 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:10.621467 12559 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:10.624871 12254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 0.
W20260810 23:32:10.636343 12596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 23:32:10.644431 12559 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
W20260810 23:32:10.636978 12596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:32:10.648841 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:10.651626 12496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e964330cf113403a8aa5b73ddd3b31d0 in term 0.
I20260810 23:32:10.654955 12218 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0; no voters:
I20260810 23:32:10.654844 12456 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0; no voters:
I20260810 23:32:10.656466 12559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:10.656721 12564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:10.657330 12564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:10.657184 12559 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:10.657907 12564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:10.658048 12559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:10.677317 12564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:10.681110 12564 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 1 election: Requested vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:10.689327 12559 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:10.693184 12559 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 1 election: Requested vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:10.684192 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0"
I20260810 23:32:10.699048 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1"
I20260810 23:32:10.701397 12254 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 681674cb24274f10b200f264f155c2d1 in current term 1: Already voted for candidate e964330cf113403a8aa5b73ddd3b31d0 in this term.
I20260810 23:32:10.704619 12496 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e964330cf113403a8aa5b73ddd3b31d0 in current term 1: Already voted for candidate 681674cb24274f10b200f264f155c2d1 in this term.
I20260810 23:32:10.609316 12410 master_runner.cc:186] Error getting master registration for 127.11.226.189:42101: OK, instance_id { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" instance_seqno: 1786404727425148 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:32:10.724371 12410 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:32:11.267086 12454 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.226.189:42101 timed out after 0.582s (SENT)
W20260810 23:32:11.308753 12215 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.226.189:42101 timed out after 0.670s (SENT) [suppressed 12 similar messages]
W20260810 23:32:11.561468 12582 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:32:11.562230 12582 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:32:11.654724 12596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:32:11.655417 12596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:32:11.868476 12454 leader_election.cc:334] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Timed out: RequestConsensusVote RPC to 127.11.226.189:42101 timed out after 0.582s (SENT)
I20260810 23:32:11.869406 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 681674cb24274f10b200f264f155c2d1; no voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:11.885380 12602 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260810 23:32:11.980579 12454 leader_election.cc:334] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Timed out: RequestConsensusVote RPC to 127.11.226.189:42101 timed out after 0.677s (SENT)
W20260810 23:32:11.989359 12215 leader_election.cc:334] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Timed out: RequestConsensusVote RPC to 127.11.226.189:42101 timed out after 0.670s (SENT)
I20260810 23:32:12.062942 12406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } attempt: 8
W20260810 23:32:12.163164 12215 leader_election.cc:334] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Timed out: RequestConsensusVote RPC to 127.11.226.189:42101 timed out after 0.733s (SENT)
I20260810 23:32:12.163904 12215 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e964330cf113403a8aa5b73ddd3b31d0; no voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1
I20260810 23:32:12.163949 12406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Bootstrap starting.
I20260810 23:32:12.181473 12605 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:32:12.198650 12406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:12.214493 12406 log.cc:824] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:12.275170 12406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: No bootstrap required, opened a new log
I20260810 23:32:12.382105 12406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:12.387254 12406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:12.387907 12406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198f534261994047a68e1d509bb2fd8b, State: Initialized, Role: FOLLOWER
I20260810 23:32:12.389926 12406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:12.412735 12406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:12.413424 12365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:12.406903 12607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:12.416781 12607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:12.415637 12365 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 0.
I20260810 23:32:12.415395 12364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
W20260810 23:32:12.418478 12365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57754 (request call id 5) took 1115 ms (client timeout 677 ms). Trace:
I20260810 23:32:12.417052 12366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b"
I20260810 23:32:12.421386 12372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b"
I20260810 23:32:12.418874 12364 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e964330cf113403a8aa5b73ddd3b31d0 in term 0.
W20260810 23:32:12.424641 12366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57754 (request call id 4) took 1142 ms (client timeout 582 ms). Trace:
W20260810 23:32:12.424494 12364 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57744 (request call id 4) took 1108 ms (client timeout 670 ms). Trace:
W20260810 23:32:12.419250 12365 rpcz_store.cc:269] 0810 23:32:11.303160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:11.303431 (+ 271us) service_pool.cc:227] Handling call
0810 23:32:12.418379 (+1114948us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 23:32:12.427832 12364 rpcz_store.cc:269] 0810 23:32:11.316282 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:11.316586 (+ 304us) service_pool.cc:227] Handling call
0810 23:32:12.424378 (+1107792us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 23:32:12.427318 12366 rpcz_store.cc:269] 0810 23:32:11.282203 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:11.282430 (+ 227us) service_pool.cc:227] Handling call
0810 23:32:12.424544 (+1142114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 23:32:12.431469 12372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57754 (request call id 3) took 1749 ms (client timeout 582 ms). Trace:
I20260810 23:32:12.431875 12373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:12.432516 12376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:12.433794 12376 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 0.
W20260810 23:32:12.434408 12376 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57754 (request call id 2) took 1818 ms (client timeout 677 ms). Trace:
W20260810 23:32:12.432243 12372 rpcz_store.cc:269] 0810 23:32:10.682354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:10.682613 (+ 259us) service_pool.cc:227] Handling call
0810 23:32:12.431400 (+1748787us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 23:32:12.434561 12373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57744 (request call id 2) took 1796 ms (client timeout 670 ms). Trace:
I20260810 23:32:12.435391 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b"
I20260810 23:32:12.442790 12362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:12.433284 12369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b"
W20260810 23:32:12.445269 12369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57744 (request call id 3) took 1748 ms (client timeout 733 ms). Trace:
W20260810 23:32:12.445976 12369 rpcz_store.cc:269] 0810 23:32:10.696901 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:10.697247 (+ 346us) service_pool.cc:227] Handling call
0810 23:32:12.445176 (+1747929us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 23:32:12.444470 12373 rpcz_store.cc:269] 0810 23:32:10.638643 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:10.638906 (+ 263us) service_pool.cc:227] Handling call
0810 23:32:12.434494 (+1795588us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 23:32:12.441462 12376 rpcz_store.cc:269] 0810 23:32:10.616499 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:10.616791 (+ 292us) service_pool.cc:227] Handling call
0810 23:32:12.434345 (+1817554us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:12.462075 12362 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e964330cf113403a8aa5b73ddd3b31d0 in term 1.
W20260810 23:32:12.463121 12362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57744 (request call id 5) took 1033 ms (client timeout 733 ms). Trace:
W20260810 23:32:12.463902 12362 rpcz_store.cc:269] 0810 23:32:11.430472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:11.430680 (+ 208us) service_pool.cc:227] Handling call
0810 23:32:12.463040 (+1032360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:12.513624 12622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:12.515044 12622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:12.525625 12622 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:12.527655 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:12.529028 12254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 1.
I20260810 23:32:12.530844 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:12.532148 12362 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 1.
I20260810 23:32:12.534276 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1; no voters:
W20260810 23:32:12.535115 12334 tablet.cc:2369] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:12.536051 12622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:32:12.536725 12622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:12.537396 12622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:32:12.555996 12622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:12.562146 12622 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 2 election: Requested vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:12.563206 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0"
I20260810 23:32:12.564502 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b"
I20260810 23:32:12.566836 12362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 1 FOLLOWER]: Advancing to term 2
W20260810 23:32:12.567515 12582 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:32:12.567860 12582 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:32:12.563858 12254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:32:12.587106 12254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 2.
I20260810 23:32:12.590754 12362 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 2.
I20260810 23:32:12.592568 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1; no voters:
I20260810 23:32:12.593670 12622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:32:12.595953 12622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 2 LEADER]: Becoming Leader. State: Replica: 681674cb24274f10b200f264f155c2d1, State: Running, Role: LEADER
I20260810 23:32:12.597218 12622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:12.621304 12630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 681674cb24274f10b200f264f155c2d1. Latest consensus state: current_term: 2 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:12.622743 12630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:12.633600 12636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:12.671051 12636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 23:32:12.673748 12633 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:32:12.674309 12633 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:32:12.724749 12362 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Refusing update from remote peer 681674cb24274f10b200f264f155c2d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 23:32:12.725293 12254 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 2 FOLLOWER]: Refusing update from remote peer 681674cb24274f10b200f264f155c2d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 23:32:12.727100 12622 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:12.728163 12630 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:12.813984 12607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 681674cb24274f10b200f264f155c2d1. Latest consensus state: current_term: 2 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:12.818836 12607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:12.869879 12645 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 681674cb24274f10b200f264f155c2d1. Latest consensus state: current_term: 2 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:12.871376 12645 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:12.871708 12630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:12.873085 12630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:12.878242 12636 catalog_manager.cc:1347] Generated new cluster ID: f4f3e41035bf41dc9042550cc0b04e2a
I20260810 23:32:12.878713 12636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:12.898540 12607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:12.899665 12607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:12.891913 12655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:12.902400 12655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:12.907395 12651 mvcc.cc:205] Tried to move back new op lower bound from 7317113785205579776 to 7317113784731336704. Current Snapshot: MvccSnapshot[applied={T|T < 7317113785205579776}]
I20260810 23:32:12.925773 12645 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:12.926739 12645 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:12.990798 12636 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:12.998626 12636 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:13.065394 12636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1: Generated new TSK 0
I20260810 23:32:13.067092 12636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:32:13.095993 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.226.129:0
--local_ip_for_outbound_sockets=127.11.226.129
--webserver_interface=127.11.226.129
--webserver_port=0
--tserver_master_addrs=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--builtin_ntp_servers=127.11.226.148:41081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260810 23:32:13.579396 12582 catalog_manager.cc:1259] Loaded cluster ID: f4f3e41035bf41dc9042550cc0b04e2a
I20260810 23:32:13.588263 12582 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: loading cluster ID for follower catalog manager: success
I20260810 23:32:13.619566 12582 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: acquiring CA information for follower catalog manager: success
I20260810 23:32:13.652745 12582 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:32:13.693567 12633 catalog_manager.cc:1259] Loaded cluster ID: f4f3e41035bf41dc9042550cc0b04e2a
I20260810 23:32:13.694180 12633 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: loading cluster ID for follower catalog manager: success
I20260810 23:32:13.739462 12633 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: acquiring CA information for follower catalog manager: success
I20260810 23:32:13.756763 12633 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 23:32:13.979089 12658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:13.980767 12658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:13.982465 12658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:14.054548 12658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:14.056676 12658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.226.129
I20260810 23:32:14.133888 12658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:41081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.226.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.226.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.226.129
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:32:14.136649 12658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:14.139896 12658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:14.243585 12666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:14.470044 12672 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:14.471004 12672 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:14.480465 12376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:14.490720 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:14.493803 12218 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e964330cf113403a8aa5b73ddd3b31d0; no voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1
I20260810 23:32:14.497160 12672 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:14.504663 12672 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 23:32:14.901221 12678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:14.901891 12678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:14.905258 12678 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:14.921700 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:14.925077 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:14.929440 12332 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b; no voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:14.933631 12678 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 23:32:15.472154 12679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:15.473145 12679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:15.476794 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:15.482069 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:15.484870 12332 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b; no voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:15.489859 12679 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:15.491966 12679 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 23:32:15.518507 12682 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:15.524668 12682 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:15.533728 12682 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:15.536165 12376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:15.537295 12376 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e964330cf113403a8aa5b73ddd3b31d0 in term 2.
I20260810 23:32:15.538827 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:15.542423 12215 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0; no voters:
I20260810 23:32:15.544047 12682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 23:32:15.544706 12682 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:15.545398 12682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:32:15.564978 12682 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:15.570084 12682 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 3 election: Requested vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:15.572854 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1"
I20260810 23:32:15.585261 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b"
I20260810 23:32:15.591538 12215 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e964330cf113403a8aa5b73ddd3b31d0; no voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1
I20260810 23:32:15.593304 12682 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260810 23:32:15.737684 12410 master_runner.cc:232] Existing masters have differing terms: 3 vs 2
I20260810 23:32:15.738310 12410 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:32:14.255357 12669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:14.248759 12667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:15.845052 12668 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 23:32:15.845240 12658 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 23:32:15.851824 12658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:15.855448 12658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:15.856972 12658 hybrid_clock.cc:648] HybridClock initialized: now 1786404735856921 us; error 73 us; skew 500 ppm
I20260810 23:32:15.857913 12658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:15.871302 12658 webserver.cc:460] Webserver started at http://127.11.226.129:37007/ using document root <none> and password file <none>
I20260810 23:32:15.872694 12658 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:15.873003 12658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:15.873595 12658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:15.879635 12658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/data/instance:
uuid: "191d120f4c7f44fc8475305a6d2e75f3"
format_stamp: "Formatted at 2026-08-10 23:32:15 on dist-test-slave-kvfs"
I20260810 23:32:15.881546 12658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/instance:
uuid: "191d120f4c7f44fc8475305a6d2e75f3"
format_stamp: "Formatted at 2026-08-10 23:32:15 on dist-test-slave-kvfs"
I20260810 23:32:15.894706 12658 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.008s
I20260810 23:32:15.907595 12695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:15.909524 12658 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.007s
I20260810 23:32:15.909953 12658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal
uuid: "191d120f4c7f44fc8475305a6d2e75f3"
format_stamp: "Formatted at 2026-08-10 23:32:15 on dist-test-slave-kvfs"
I20260810 23:32:15.910571 12658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:15.990859 12254 raft_consensus.cc:1226] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 3 FOLLOWER]: Rejecting Update request from peer 681674cb24274f10b200f264f155c2d1 for earlier term 2. Current term is 3. Ops: []
I20260810 23:32:15.993249 12685 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 23:32:15.994768 12683 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 23:32:15.995313 12683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 681674cb24274f10b200f264f155c2d1, State: Running, Role: LEADER
I20260810 23:32:15.995997 12658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:15.997216 12683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:15.998842 12658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:15.999254 12683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:32:15.999605 12658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:16.003899 12658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:16.012820 12658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:16.013164 12658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:16.013496 12658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:16.013854 12658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:16.040391 12712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:16.041606 12712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:16.044975 12712 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:16.052498 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:16.054060 12254 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 198f534261994047a68e1d509bb2fd8b in current term 3: Already voted for candidate e964330cf113403a8aa5b73ddd3b31d0 in this term.
I20260810 23:32:16.055727 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:16.056985 12496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 198f534261994047a68e1d509bb2fd8b in term 3.
I20260810 23:32:16.061689 12332 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1; no voters: e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:16.064386 12712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 23:32:16.065189 12712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:16.065819 12712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:32:16.081508 12712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:16.086017 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0"
I20260810 23:32:16.087186 12254 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 198f534261994047a68e1d509bb2fd8b in current term 3: Already voted for candidate e964330cf113403a8aa5b73ddd3b31d0 in this term.
I20260810 23:32:16.091639 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1"
I20260810 23:32:16.093618 12712 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 3 election: Requested vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:16.111497 12496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 198f534261994047a68e1d509bb2fd8b in term 3.
I20260810 23:32:16.116011 12332 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1; no voters: e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:16.118144 12712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:32:16.124483 12712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 3 LEADER]: Becoming Leader. State: Replica: 198f534261994047a68e1d509bb2fd8b, State: Running, Role: LEADER
I20260810 23:32:16.127161 12712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:16.142774 12724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 198f534261994047a68e1d509bb2fd8b. Latest consensus state: current_term: 3 leader_uuid: "198f534261994047a68e1d509bb2fd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:16.144846 12724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:16.149138 12729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:16.165588 12729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:16.176714 12729 catalog_manager.cc:1259] Loaded cluster ID: f4f3e41035bf41dc9042550cc0b04e2a
I20260810 23:32:16.177419 12729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:16.190131 12729 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:16.200560 12729 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Loaded TSK: 0
I20260810 23:32:16.204248 12729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:32:16.267455 12762 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:16.268651 12762 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:16.275784 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:16.279614 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:16.280208 12762 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:16.280653 12496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e964330cf113403a8aa5b73ddd3b31d0 in term 3.
I20260810 23:32:16.282704 12218 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0; no voters: 198f534261994047a68e1d509bb2fd8b
I20260810 23:32:16.285075 12762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 23:32:16.285739 12762 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:16.286644 12762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 3 FOLLOWER]: Advancing to term 4
I20260810 23:32:16.301764 12762 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:16.308748 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b"
I20260810 23:32:16.311196 12762 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 4 election: Requested vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:16.308743 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1"
I20260810 23:32:16.313109 12496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 3 FOLLOWER]: Advancing to term 4
I20260810 23:32:16.324993 12496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e964330cf113403a8aa5b73ddd3b31d0 in term 4.
I20260810 23:32:16.327560 12218 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0; no voters: 198f534261994047a68e1d509bb2fd8b
I20260810 23:32:16.330088 12762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 4 FOLLOWER]: Leader election won for term 4
I20260810 23:32:16.336195 12762 raft_consensus.cc:686] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 4 LEADER]: Becoming Leader. State: Replica: e964330cf113403a8aa5b73ddd3b31d0, State: Running, Role: LEADER
I20260810 23:32:16.338634 12762 consensus_queue.cc:237] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:16.350412 12774 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e964330cf113403a8aa5b73ddd3b31d0. Latest consensus state: current_term: 4 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:16.351631 12774 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:16.354753 12780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:16.370034 12780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:16.378710 12780 catalog_manager.cc:1259] Loaded cluster ID: f4f3e41035bf41dc9042550cc0b04e2a
I20260810 23:32:16.379384 12780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:16.389348 12780 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:16.397059 12780 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Loaded TSK: 0
I20260810 23:32:16.400435 12780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:32:16.640400 12254 raft_consensus.cc:1226] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 4 LEADER]: Rejecting Update request from peer 198f534261994047a68e1d509bb2fd8b for earlier term 3. Current term is 4. Ops: []
I20260810 23:32:16.643949 12724 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 23:32:16.645991 12724 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [LEADER]: Peer e964330cf113403a8aa5b73ddd3b31d0 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260810 23:32:16.646886 12868 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 3 LEADER]: Stepping down as leader of term 3
I20260810 23:32:16.647583 12868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 198f534261994047a68e1d509bb2fd8b, State: Running, Role: LEADER
I20260810 23:32:16.649362 12868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:16.651943 12868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 3 FOLLOWER]: Advancing to term 4
I20260810 23:32:16.673454 12596 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:32:16.683115 12658 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.129:46059
I20260810 23:32:16.683404 12876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.129:46059 every 8 connection(s)
I20260810 23:32:16.686764 12658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/data/info.pb
I20260810 23:32:16.697888 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 12658
I20260810 23:32:16.698578 12170 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/instance
I20260810 23:32:16.722607 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.226.130:0
--local_ip_for_outbound_sockets=127.11.226.130
--webserver_interface=127.11.226.130
--webserver_port=0
--tserver_master_addrs=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--builtin_ntp_servers=127.11.226.148:41081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260810 23:32:16.793255 12879 heartbeater.cc:347] Connected to a master server at 127.11.226.190:43533
I20260810 23:32:16.794226 12879 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:16.800235 12878 heartbeater.cc:347] Connected to a master server at 127.11.226.189:42101
I20260810 23:32:16.800792 12878 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:16.802343 12877 heartbeater.cc:347] Connected to a master server at 127.11.226.188:45477
I20260810 23:32:16.802909 12877 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:16.803143 12878 heartbeater.cc:511] Master 127.11.226.189:42101 requested a full tablet report, sending...
I20260810 23:32:16.804659 12877 heartbeater.cc:511] Master 127.11.226.188:45477 requested a full tablet report, sending...
I20260810 23:32:16.806751 12879 heartbeater.cc:511] Master 127.11.226.190:43533 requested a full tablet report, sending...
I20260810 23:32:16.810738 12482 ts_manager.cc:194] Registered new tserver with Master: 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:16.810952 12357 ts_manager.cc:194] Registered new tserver with Master: 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:16.811474 12240 ts_manager.cc:194] Registered new tserver with Master: 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:16.815848 12240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.226.129:56187
I20260810 23:32:16.919570 12496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 4 FOLLOWER]: Refusing update from remote peer e964330cf113403a8aa5b73ddd3b31d0: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260810 23:32:16.921573 12899 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 23:32:16.934767 12900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e964330cf113403a8aa5b73ddd3b31d0. Latest consensus state: current_term: 4 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:16.936524 12900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:16.951855 12900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:16.952503 12362 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 4 FOLLOWER]: Refusing update from remote peer e964330cf113403a8aa5b73ddd3b31d0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260810 23:32:16.953348 12362 pending_rounds.cc:76] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Aborting all ops after (but not including) 4
I20260810 23:32:16.953850 12362 pending_rounds.cc:97] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260810 23:32:16.954089 12900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:16.954416 12362 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 23:32:16.961793 12899 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 23:32:16.966125 12899 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:16.967527 12899 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:16.983736 12724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:16.984922 12724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:16.985077 12868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e964330cf113403a8aa5b73ddd3b31d0. Latest consensus state: current_term: 4 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:16.986169 12868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:17.322548 12879 heartbeater.cc:503] Master 127.11.226.190:43533 was elected leader, sending a full tablet report...
I20260810 23:32:17.413028 12899 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:17.414554 12899 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: LEADER
W20260810 23:32:17.427318 12897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:17.428174 12897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:17.428874 12897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:17.466951 12897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:17.468340 12897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.226.130
I20260810 23:32:17.509766 12897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:41081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.226.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.226.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.226.130
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:32:17.511670 12897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:17.513971 12897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:17.542158 12921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:17.913482 12928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e964330cf113403a8aa5b73ddd3b31d0)
I20260810 23:32:17.914251 12928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:17.919103 12928 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:17.920466 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:17.921027 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:17.921434 12362 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 4.
I20260810 23:32:17.922791 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1; no voters:
I20260810 23:32:17.924897 12928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 23:32:17.925796 12928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e964330cf113403a8aa5b73ddd3b31d0)
I20260810 23:32:17.926620 12928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 4 FOLLOWER]: Advancing to term 5
I20260810 23:32:17.922000 12929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e964330cf113403a8aa5b73ddd3b31d0)
I20260810 23:32:17.928577 12929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:17.936532 12929 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:17.938686 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:17.940187 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:17.948562 12928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:17.951786 12496 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 198f534261994047a68e1d509bb2fd8b in current term 5: Already voted for candidate 681674cb24274f10b200f264f155c2d1 in this term.
I20260810 23:32:17.952636 12928 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 5 election: Requested vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:17.954737 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0"
I20260810 23:32:17.955427 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b"
I20260810 23:32:17.957949 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 681674cb24274f10b200f264f155c2d1; no voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:17.959901 12928 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260810 23:32:17.961606 12332 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b; no voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:17.963243 12929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 4 FOLLOWER]: Advancing to term 5
I20260810 23:32:17.981992 12929 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 23:32:17.996767 12496 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 5 FOLLOWER]: Rejecting Update request from peer e964330cf113403a8aa5b73ddd3b31d0 for earlier term 4. Current term is 5. Ops: []
I20260810 23:32:18.000502 12899 consensus_queue.cc:1046] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 23:32:18.002635 12930 raft_consensus.cc:3032] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 4 LEADER]: Stepping down as leader of term 4
I20260810 23:32:18.002979 12930 raft_consensus.cc:729] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e964330cf113403a8aa5b73ddd3b31d0, State: Running, Role: LEADER
I20260810 23:32:18.003712 12930 consensus_queue.cc:260] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:18.005103 12930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 4 FOLLOWER]: Advancing to term 5
I20260810 23:32:18.506597 12933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:18.507848 12933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:18.511871 12933 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:18.513008 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:18.513514 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:18.513787 12254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 5.
I20260810 23:32:18.514238 12362 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 5.
I20260810 23:32:18.515767 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1; no voters:
I20260810 23:32:18.516986 12933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 23:32:18.517469 12933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:18.517875 12933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 5 FOLLOWER]: Advancing to term 6
I20260810 23:32:18.526007 12933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:18.528158 12933 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 6 election: Requested vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:18.529655 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0"
I20260810 23:32:18.529724 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b"
I20260810 23:32:18.530346 12362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 5 FOLLOWER]: Advancing to term 6
I20260810 23:32:18.530345 12254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 5 FOLLOWER]: Advancing to term 6
I20260810 23:32:18.538700 12254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 6.
I20260810 23:32:18.538700 12362 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 6.
I20260810 23:32:18.540274 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1; no voters:
I20260810 23:32:18.541254 12933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 6 FOLLOWER]: Leader election won for term 6
I20260810 23:32:18.542896 12933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 6 LEADER]: Becoming Leader. State: Replica: 681674cb24274f10b200f264f155c2d1, State: Running, Role: LEADER
I20260810 23:32:18.543859 12933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:18.548903 12935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 681674cb24274f10b200f264f155c2d1. Latest consensus state: current_term: 6 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:18.550024 12935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:18.552491 12937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:18.566097 12937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:18.573177 12937 catalog_manager.cc:1259] Loaded cluster ID: f4f3e41035bf41dc9042550cc0b04e2a
I20260810 23:32:18.573683 12937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:18.582895 12937 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:18.590006 12937 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1: Loaded TSK: 0
I20260810 23:32:18.592803 12937 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:32:17.542479 12919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:17.542794 12924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:17.545770 12897 server_base.cc:1034] running on GCE node
I20260810 23:32:18.871645 12897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:18.874800 12897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:18.876327 12897 hybrid_clock.cc:648] HybridClock initialized: now 1786404738876281 us; error 83 us; skew 500 ppm
I20260810 23:32:18.877470 12897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:18.886237 12897 webserver.cc:460] Webserver started at http://127.11.226.130:40239/ using document root <none> and password file <none>
I20260810 23:32:18.887518 12897 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:18.887826 12897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:18.888630 12897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:18.895552 12897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/data/instance:
uuid: "f26517150f1c480a877f15aec71d020b"
format_stamp: "Formatted at 2026-08-10 23:32:18 on dist-test-slave-kvfs"
I20260810 23:32:18.897287 12897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/instance:
uuid: "f26517150f1c480a877f15aec71d020b"
format_stamp: "Formatted at 2026-08-10 23:32:18 on dist-test-slave-kvfs"
I20260810 23:32:18.907363 12897 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.001s
I20260810 23:32:18.914852 12945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:18.916409 12897 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20260810 23:32:18.916827 12897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal
uuid: "f26517150f1c480a877f15aec71d020b"
format_stamp: "Formatted at 2026-08-10 23:32:18 on dist-test-slave-kvfs"
I20260810 23:32:18.917328 12897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:18.925158 12362 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 6 FOLLOWER]: Refusing update from remote peer 681674cb24274f10b200f264f155c2d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260810 23:32:18.927228 12935 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 23:32:18.938002 12946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 681674cb24274f10b200f264f155c2d1. Latest consensus state: current_term: 6 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:18.939085 12946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:18.949656 12946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:18.950886 12946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:18.959303 12254 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 6 FOLLOWER]: Refusing update from remote peer 681674cb24274f10b200f264f155c2d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260810 23:32:18.962158 12957 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 23:32:18.964206 12957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:18.965026 12957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:18.976437 12967 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:18.976963 12963 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 681674cb24274f10b200f264f155c2d1. Latest consensus state: current_term: 6 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:18.977555 12967 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:18.977818 12963 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:18.995632 12897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:18.997617 12897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:18.998328 12897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:19.001569 12897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:19.007153 12897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:19.007512 12897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:19.007839 12897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:19.008070 12897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:19.337613 12877 heartbeater.cc:503] Master 127.11.226.188:45477 was elected leader, sending a full tablet report...
I20260810 23:32:19.469503 12963 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:19.470181 12963 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:19.472786 12963 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:19.474526 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:19.475468 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:19.477216 12218 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e964330cf113403a8aa5b73ddd3b31d0; no voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1
I20260810 23:32:19.477969 12963 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:32:19.575466 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:19.576552 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:19.623733 12897 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.130:42079
I20260810 23:32:19.624166 13138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.130:42079 every 8 connection(s)
I20260810 23:32:19.627813 12897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/data/info.pb
I20260810 23:32:19.637434 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 12897
I20260810 23:32:19.638098 12170 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/instance
I20260810 23:32:19.691239 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.226.131:0
--local_ip_for_outbound_sockets=127.11.226.131
--webserver_interface=127.11.226.131
--webserver_port=0
--tserver_master_addrs=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--builtin_ntp_servers=127.11.226.148:41081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260810 23:32:19.812384 13139 heartbeater.cc:347] Connected to a master server at 127.11.226.188:45477
I20260810 23:32:19.813261 13139 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:19.815686 13139 heartbeater.cc:511] Master 127.11.226.188:45477 requested a full tablet report, sending...
I20260810 23:32:19.822154 12482 ts_manager.cc:194] Registered new tserver with Master: f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:19.826725 13141 heartbeater.cc:347] Connected to a master server at 127.11.226.189:42101
I20260810 23:32:19.827265 13141 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:19.829669 13141 heartbeater.cc:511] Master 127.11.226.189:42101 requested a full tablet report, sending...
I20260810 23:32:19.830652 12482 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.226.130:55893
I20260810 23:32:19.835016 12357 ts_manager.cc:194] Registered new tserver with Master: f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:19.872991 13146 heartbeater.cc:347] Connected to a master server at 127.11.226.190:43533
I20260810 23:32:19.873414 13158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:19.873698 13146 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:19.874225 13158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:19.875262 13146 heartbeater.cc:511] Master 127.11.226.190:43533 requested a full tablet report, sending...
I20260810 23:32:19.878046 13158 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:19.880285 12240 ts_manager.cc:194] Registered new tserver with Master: f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:19.880676 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:19.882551 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:19.885421 12332 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b; no voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:19.886941 13158 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:32:20.341641 13139 heartbeater.cc:503] Master 127.11.226.188:45477 was elected leader, sending a full tablet report...
W20260810 23:32:20.376319 13149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:20.377156 13149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:20.377933 13149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:20.418368 13149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:20.419651 13149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.226.131
I20260810 23:32:20.437355 13165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:20.438117 13165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:20.441318 13165 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:20.443413 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:20.445710 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:20.447976 12332 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b; no voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:20.450155 13165 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:32:20.471661 13149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:41081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.226.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.226.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.226.131
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:32:20.473671 13149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:20.476224 13149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:32:20.505430 13166 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:20.511495 13166 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:20.520282 13166 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:20.522549 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:20.525250 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:20.529342 12215 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e964330cf113403a8aa5b73ddd3b31d0; no voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1
I20260810 23:32:20.538007 13166 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 23:32:20.545390 13169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:21.008998 13178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:21.011054 13178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:21.018989 13178 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:21.021438 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:21.022607 12254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 198f534261994047a68e1d509bb2fd8b in term 6.
I20260810 23:32:21.026945 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:21.030007 12332 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0; no voters:
I20260810 23:32:21.038306 13178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260810 23:32:21.038764 13178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:21.039252 13178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 6 FOLLOWER]: Advancing to term 7
I20260810 23:32:21.050464 13178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:21.055859 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0"
I20260810 23:32:21.057224 12254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 6 FOLLOWER]: Advancing to term 7
I20260810 23:32:21.058952 13178 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 7 election: Requested vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:21.060626 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1"
I20260810 23:32:21.075462 12254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 198f534261994047a68e1d509bb2fd8b in term 7.
I20260810 23:32:21.078018 12332 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0; no voters: 681674cb24274f10b200f264f155c2d1
I20260810 23:32:21.080456 13178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 7 FOLLOWER]: Leader election won for term 7
I20260810 23:32:21.096729 13178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 7 LEADER]: Becoming Leader. State: Replica: 198f534261994047a68e1d509bb2fd8b, State: Running, Role: LEADER
I20260810 23:32:21.098289 13178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:21.111277 12254 raft_consensus.cc:1226] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 7 FOLLOWER]: Rejecting Update request from peer 681674cb24274f10b200f264f155c2d1 for earlier term 6. Current term is 7. Ops: []
I20260810 23:32:21.117863 13181 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260810 23:32:21.122547 13181 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 6 LEADER]: Stepping down as leader of term 6
I20260810 23:32:21.125264 13181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 681674cb24274f10b200f264f155c2d1, State: Running, Role: LEADER
I20260810 23:32:21.125494 13183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 198f534261994047a68e1d509bb2fd8b. Latest consensus state: current_term: 7 leader_uuid: "198f534261994047a68e1d509bb2fd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:21.126443 13183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:21.127636 12362 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 7 LEADER]: Rejecting Update request from peer 681674cb24274f10b200f264f155c2d1 for earlier term 6. Current term is 7. Ops: []
I20260810 23:32:21.127727 13181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:21.128954 13181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 6 FOLLOWER]: Advancing to term 7
I20260810 23:32:21.141081 13189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:21.151773 13189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:21.161681 13189 catalog_manager.cc:1259] Loaded cluster ID: f4f3e41035bf41dc9042550cc0b04e2a
I20260810 23:32:21.162559 13189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:21.174329 13189 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:21.184638 13189 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Loaded TSK: 0
I20260810 23:32:21.187803 13189 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:32:19.732178 12181 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.ConnectToMaster -> {remote=127.11.226.188:45477, user_credentials={real_user=slave}} blocked reactor thread for 45871.3us
I20260810 23:32:21.554410 13181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:21.555162 13181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:21.557955 13181 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:21.559741 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:21.560770 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:21.560832 12254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 7.
I20260810 23:32:21.564651 12456 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0; no voters: 198f534261994047a68e1d509bb2fd8b
I20260810 23:32:21.566222 13181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260810 23:32:21.566632 13181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:21.567028 13181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 7 FOLLOWER]: Advancing to term 8
I20260810 23:32:21.577903 13181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:21.583678 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0"
I20260810 23:32:21.585402 12254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 7 FOLLOWER]: Advancing to term 8
I20260810 23:32:21.586287 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b"
I20260810 23:32:21.587518 13181 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 8 election: Requested vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:21.598140 12254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 8.
I20260810 23:32:21.599910 12456 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0; no voters: 198f534261994047a68e1d509bb2fd8b
I20260810 23:32:21.601152 13181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 8 FOLLOWER]: Leader election won for term 8
I20260810 23:32:21.602063 13181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 8 LEADER]: Becoming Leader. State: Replica: 681674cb24274f10b200f264f155c2d1, State: Running, Role: LEADER
I20260810 23:32:21.603951 13181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:21.616257 13187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 681674cb24274f10b200f264f155c2d1. Latest consensus state: current_term: 8 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:21.617295 13187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:21.618244 12254 raft_consensus.cc:1226] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 8 FOLLOWER]: Rejecting Update request from peer 198f534261994047a68e1d509bb2fd8b for earlier term 7. Current term is 8. Ops: []
I20260810 23:32:21.621848 13183 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260810 23:32:21.624218 13178 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 7 LEADER]: Stepping down as leader of term 7
I20260810 23:32:21.624946 13178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 198f534261994047a68e1d509bb2fd8b, State: Running, Role: LEADER
I20260810 23:32:21.626643 13178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:21.629945 13178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 7 FOLLOWER]: Advancing to term 8
I20260810 23:32:21.642452 13193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:21.666397 13193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:21.675514 13193 catalog_manager.cc:1259] Loaded cluster ID: f4f3e41035bf41dc9042550cc0b04e2a
I20260810 23:32:21.676240 13193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:21.688828 13193 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:21.698146 13193 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1: Loaded TSK: 0
I20260810 23:32:21.701264 13193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:32:21.865702 13141 heartbeater.cc:503] Master 127.11.226.189:42101 was elected leader, sending a full tablet report...
I20260810 23:32:21.895015 12878 heartbeater.cc:503] Master 127.11.226.189:42101 was elected leader, sending a full tablet report...
I20260810 23:32:22.014395 12362 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 8 FOLLOWER]: Refusing update from remote peer 681674cb24274f10b200f264f155c2d1: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20260810 23:32:22.014727 12362 pending_rounds.cc:76] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Aborting all ops after (but not including) 6
I20260810 23:32:22.014991 12362 pending_rounds.cc:97] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260810 23:32:22.015254 12362 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 23:32:22.015803 13178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:22.016543 13178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:22.017119 13187 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 23:32:22.021587 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:22.021723 13183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 681674cb24274f10b200f264f155c2d1. Latest consensus state: current_term: 8 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:22.022508 13183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:22.022493 12254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 198f534261994047a68e1d509bb2fd8b in term 8.
I20260810 23:32:22.024468 13178 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:22.024204 12332 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0; no voters:
I20260810 23:32:22.025394 13178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260810 23:32:22.025818 13178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:22.026312 13178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 8 FOLLOWER]: Advancing to term 9
I20260810 23:32:22.026731 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:22.034713 13181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:22.037010 12254 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 8 FOLLOWER]: Refusing update from remote peer 681674cb24274f10b200f264f155c2d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260810 23:32:22.038098 13181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:22.039541 13181 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260810 23:32:22.041424 13178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:22.045883 12362 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 9 FOLLOWER]: Rejecting Update request from peer 681674cb24274f10b200f264f155c2d1 for earlier term 8. Current term is 9. Ops: []
I20260810 23:32:22.053068 12252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0"
I20260810 23:32:22.053957 13181 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260810 23:32:22.055962 13187 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 8 LEADER]: Stepping down as leader of term 8
I20260810 23:32:22.056453 13187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 681674cb24274f10b200f264f155c2d1, State: Running, Role: LEADER
I20260810 23:32:22.057894 13187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:22.058986 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1"
I20260810 23:32:22.059372 13187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 8 FOLLOWER]: Advancing to term 9
I20260810 23:32:22.060181 13178 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 9 election: Requested vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:22.073136 13198 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:22.072958 13195 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 681674cb24274f10b200f264f155c2d1. Latest consensus state: current_term: 8 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:22.074328 13195 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:22.075115 13198 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:22.089574 12496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 198f534261994047a68e1d509bb2fd8b in term 9.
I20260810 23:32:22.091931 12332 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1; no voters: e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:22.093190 13178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 9 FOLLOWER]: Leader election won for term 9
I20260810 23:32:22.093940 13178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 9 LEADER]: Becoming Leader. State: Replica: 198f534261994047a68e1d509bb2fd8b, State: Running, Role: LEADER
I20260810 23:32:22.095118 13178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:22.099148 13183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 198f534261994047a68e1d509bb2fd8b. Latest consensus state: current_term: 9 leader_uuid: "198f534261994047a68e1d509bb2fd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:22.100003 13183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:22.102089 13199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:22.111912 13199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:22.116566 13199 catalog_manager.cc:1259] Loaded cluster ID: f4f3e41035bf41dc9042550cc0b04e2a
I20260810 23:32:22.117039 13199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:22.122324 13199 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:22.126878 13199 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Loaded TSK: 0
I20260810 23:32:22.128602 13199 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:32:20.571429 13175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:20.554909 13171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:22.278654 13149 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260810 23:32:22.278913 13173 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 23:32:22.284389 13149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:22.288797 13149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:22.290365 13149 hybrid_clock.cc:648] HybridClock initialized: now 1786404742290319 us; error 47 us; skew 500 ppm
I20260810 23:32:22.291703 13149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:22.305466 13149 webserver.cc:460] Webserver started at http://127.11.226.131:33379/ using document root <none> and password file <none>
I20260810 23:32:22.307091 13149 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:22.307403 13149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:22.308317 13149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:22.315479 13149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/data/instance:
uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
format_stamp: "Formatted at 2026-08-10 23:32:22 on dist-test-slave-kvfs"
I20260810 23:32:22.317386 13149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/instance:
uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
format_stamp: "Formatted at 2026-08-10 23:32:22 on dist-test-slave-kvfs"
I20260810 23:32:22.329543 13149 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.006s sys 0.005s
I20260810 23:32:22.340476 13209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:22.342722 13149 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.006s
I20260810 23:32:22.343403 13149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal
uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
format_stamp: "Formatted at 2026-08-10 23:32:22 on dist-test-slave-kvfs"
I20260810 23:32:22.343843 13149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:22.376107 13139 heartbeater.cc:503] Master 127.11.226.188:45477 was elected leader, sending a full tablet report...
I20260810 23:32:22.399978 13149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:22.403559 13149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:22.403563 12877 heartbeater.cc:503] Master 127.11.226.188:45477 was elected leader, sending a full tablet report...
I20260810 23:32:22.404585 13149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:22.413626 13149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:22.422436 13149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:22.422744 13149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:22.423113 13149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:22.423300 13149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:22.550433 13198 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 681674cb24274f10b200f264f155c2d1)
I20260810 23:32:22.551038 13198 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:22.553449 13198 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:22.555558 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:22.556259 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:22.557749 12496 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e964330cf113403a8aa5b73ddd3b31d0 in current term 9: Already voted for candidate 198f534261994047a68e1d509bb2fd8b in this term.
I20260810 23:32:22.559649 12218 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e964330cf113403a8aa5b73ddd3b31d0; no voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1
I20260810 23:32:22.561322 13198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 8 FOLLOWER]: Advancing to term 9
I20260810 23:32:22.576620 13198 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260810 23:32:22.615662 12496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 9 FOLLOWER]: Refusing update from remote peer 198f534261994047a68e1d509bb2fd8b: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260810 23:32:22.617772 13284 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 23:32:22.625953 13293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 198f534261994047a68e1d509bb2fd8b. Latest consensus state: current_term: 9 leader_uuid: "198f534261994047a68e1d509bb2fd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:22.626859 13293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:22.647642 13284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "198f534261994047a68e1d509bb2fd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:22.649132 13284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:22.648365 13297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "198f534261994047a68e1d509bb2fd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:22.650044 13297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:22.654883 12254 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 9 FOLLOWER]: Refusing update from remote peer 198f534261994047a68e1d509bb2fd8b: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260810 23:32:22.656908 13293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "198f534261994047a68e1d509bb2fd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:22.658232 13293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:22.662052 13284 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 23:32:22.687557 13324 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "198f534261994047a68e1d509bb2fd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:22.688256 13198 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 198f534261994047a68e1d509bb2fd8b. Latest consensus state: current_term: 9 leader_uuid: "198f534261994047a68e1d509bb2fd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:22.690526 13198 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:22.691236 13324 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:22.879004 13141 heartbeater.cc:503] Master 127.11.226.189:42101 was elected leader, sending a full tablet report...
I20260810 23:32:22.906809 12878 heartbeater.cc:503] Master 127.11.226.189:42101 was elected leader, sending a full tablet report...
I20260810 23:32:23.001260 13149 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.131:33807
I20260810 23:32:23.001398 13402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.131:33807 every 8 connection(s)
I20260810 23:32:23.005203 13149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/data/info.pb
I20260810 23:32:23.013475 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 13149
I20260810 23:32:23.015105 12170 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/instance
I20260810 23:32:23.141597 13406 heartbeater.cc:347] Connected to a master server at 127.11.226.189:42101
I20260810 23:32:23.142339 13406 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:23.144517 13406 heartbeater.cc:511] Master 127.11.226.189:42101 requested a full tablet report, sending...
I20260810 23:32:23.148506 12357 ts_manager.cc:194] Registered new tserver with Master: f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:23.151162 13404 heartbeater.cc:347] Connected to a master server at 127.11.226.188:45477
I20260810 23:32:23.151667 13404 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:23.152498 12357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.226.131:39701
I20260810 23:32:23.152855 13404 heartbeater.cc:511] Master 127.11.226.188:45477 requested a full tablet report, sending...
I20260810 23:32:23.156643 12482 ts_manager.cc:194] Registered new tserver with Master: f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:23.168383 13324 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 198f534261994047a68e1d509bb2fd8b)
I20260810 23:32:23.169656 13324 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:23.170715 13409 heartbeater.cc:347] Connected to a master server at 127.11.226.190:43533
I20260810 23:32:23.171157 13409 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:23.172093 13409 heartbeater.cc:511] Master 127.11.226.190:43533 requested a full tablet report, sending...
I20260810 23:32:23.174853 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:32:23.176849 12240 ts_manager.cc:194] Registered new tserver with Master: f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:23.180109 13324 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:23.184116 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:23.187455 12218 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e964330cf113403a8aa5b73ddd3b31d0; no voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1
I20260810 23:32:23.188576 13324 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260810 23:32:23.194451 12170 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:32:23.207756 13284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "198f534261994047a68e1d509bb2fd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:23.208709 13284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:23.336510 12170 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu with pid 12295
W20260810 23:32:23.397006 12954 connection.cc:535] client connection to 127.11.226.189:42101 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:32:23.400908 13141 heartbeater.cc:644] Failed to heartbeat to 127.11.226.189:42101 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:32:23.401043 13424 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.226.189:42101)
W20260810 23:32:23.426321 12878 heartbeater.cc:644] Failed to heartbeat to 127.11.226.189:42101 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:23.561641 13461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 198f534261994047a68e1d509bb2fd8b)
I20260810 23:32:23.562197 13461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:23.564201 13461 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:23.565389 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
W20260810 23:32:23.567304 12454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:32:23.570822 12454 leader_election.cc:334] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:23.571269 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 681674cb24274f10b200f264f155c2d1; no voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:23.572008 13461 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260810 23:32:23.662464 13406 heartbeater.cc:503] Master 127.11.226.189:42101 was elected leader, sending a full tablet report...
W20260810 23:32:23.665503 13406 heartbeater.cc:644] Failed to heartbeat to 127.11.226.189:42101 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:23.712836 13467 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 198f534261994047a68e1d509bb2fd8b)
I20260810 23:32:23.713397 13467 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:23.715294 13467 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:23.716917 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:32:23.717689 12496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e964330cf113403a8aa5b73ddd3b31d0 in term 9.
W20260810 23:32:23.718058 12215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260810 23:32:23.719055 12218 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0; no voters:
I20260810 23:32:23.719985 13467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260810 23:32:23.720396 13467 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 198f534261994047a68e1d509bb2fd8b)
I20260810 23:32:23.721014 13467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 9 FOLLOWER]: Advancing to term 10
W20260810 23:32:23.723682 12215 leader_election.cc:334] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:23.731456 13467 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:23.733533 13467 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 10 election: Requested vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:32:23.735137 12496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1"
I20260810 23:32:23.735922 12496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 9 FOLLOWER]: Advancing to term 10
W20260810 23:32:23.738160 12215 leader_election.cc:334] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:23.743687 12496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e964330cf113403a8aa5b73ddd3b31d0 in term 10.
I20260810 23:32:23.744973 12218 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0; no voters: 198f534261994047a68e1d509bb2fd8b
I20260810 23:32:23.745965 13467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 10 FOLLOWER]: Leader election won for term 10
I20260810 23:32:23.747061 13467 raft_consensus.cc:686] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 10 LEADER]: Becoming Leader. State: Replica: e964330cf113403a8aa5b73ddd3b31d0, State: Running, Role: LEADER
I20260810 23:32:23.748116 13467 consensus_queue.cc:237] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:23.753239 13470 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e964330cf113403a8aa5b73ddd3b31d0. Latest consensus state: current_term: 10 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:23.754442 13470 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:23.756806 13472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:23.766914 13472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:23.770815 13472 catalog_manager.cc:1259] Loaded cluster ID: f4f3e41035bf41dc9042550cc0b04e2a
I20260810 23:32:23.771202 13472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:23.776481 13472 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:23.779973 13472 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Loaded TSK: 0
I20260810 23:32:23.781477 13472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:32:23.803701 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_68b96e520f1f4c3aa45f927d352a07f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:23.804304 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_c2338921946342ce9c9485798a60bc96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:23.803923 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_a1b885ba527e4289b12292d38c3e862c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:23.804689 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_bd37722453474b9890ef79ede94e3466"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:23.809372 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68b96e520f1f4c3aa45f927d352a07f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:23.809286 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1b885ba527e4289b12292d38c3e862c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:23.809355 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2338921946342ce9c9485798a60bc96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:23.809319 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd37722453474b9890ef79ede94e3466 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:23.832401 12496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Refusing update from remote peer e964330cf113403a8aa5b73ddd3b31d0: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260810 23:32:23.834540 13467 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260810 23:32:23.835810 12215 consensus_peers.cc:487] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 -> Peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Couldn't send request to peer 198f534261994047a68e1d509bb2fd8b. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:23.846856 13461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e964330cf113403a8aa5b73ddd3b31d0. Latest consensus state: current_term: 10 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:23.847823 13461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:23.867122 13467 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:23.868229 13467 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:23.874500 12235 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_bd37722453474b9890ef79ede94e3466" } new_table_name: "default.Table_e493da49dc464e528ff57f28dc05bf37" modify_external_catalogs: true
I20260810 23:32:23.875559 12237 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_a1b885ba527e4289b12292d38c3e862c" } new_table_name: "default.Table_8b5a278156ad4137bf0ea955a9bb88cb" modify_external_catalogs: true
I20260810 23:32:23.880831 13461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:32:23.881788 13461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:23.900235 12236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_68b96e520f1f4c3aa45f927d352a07f7" } modify_external_catalogs: true
I20260810 23:32:23.901592 12236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_68b96e520f1f4c3aa45f927d352a07f7" } modify_external_catalogs: true
I20260810 23:32:23.905892 12240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_c2338921946342ce9c9485798a60bc96" } modify_external_catalogs: true
I20260810 23:32:23.906580 12240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_c2338921946342ce9c9485798a60bc96" } modify_external_catalogs: true
W20260810 23:32:23.906808 12237 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee5e80d295c94a24854747e468d54de6 has no consensus state
W20260810 23:32:23.953053 12235 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3ee96e60575425eb3e98ae1ab461180 has no consensus state
I20260810 23:32:23.960296 12240 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 000f124f17eb4a94bf0f2a7dec188c69
W20260810 23:32:23.964581 12218 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet ee5e80d295c94a24854747e468d54de6 (table default.Table_8b5a278156ad4137bf0ea955a9bb88cb [id=c4b2684e4dd74e0f9170cffcd54c2cb9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ee5e80d295c94a24854747e468d54de6 has no consensus state
I20260810 23:32:23.967150 12236 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet f098d1ebfc3b4baeba78350ada912f08
I20260810 23:32:24.054874 12770 tablet_service.cc:1467] Processing CreateTablet for tablet ee5e80d295c94a24854747e468d54de6 (DEFAULT_TABLE table=default.Table_8b5a278156ad4137bf0ea955a9bb88cb [id=c4b2684e4dd74e0f9170cffcd54c2cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.058290 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5e80d295c94a24854747e468d54de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.060856 13031 tablet_service.cc:1467] Processing CreateTablet for tablet ee5e80d295c94a24854747e468d54de6 (DEFAULT_TABLE table=default.Table_8b5a278156ad4137bf0ea955a9bb88cb [id=c4b2684e4dd74e0f9170cffcd54c2cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.060863 13032 tablet_service.cc:1467] Processing CreateTablet for tablet b3ee96e60575425eb3e98ae1ab461180 (DEFAULT_TABLE table=default.Table_e493da49dc464e528ff57f28dc05bf37 [id=d24331d84ad04768ab54058120b7e72f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.063433 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5e80d295c94a24854747e468d54de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.065726 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ee96e60575425eb3e98ae1ab461180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.065102 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_4ba2f23aa6fb42398071dfe0239679f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:24.067320 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ba2f23aa6fb42398071dfe0239679f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:24.053233 12772 tablet_service.cc:1467] Processing CreateTablet for tablet b3ee96e60575425eb3e98ae1ab461180 (DEFAULT_TABLE table=default.Table_e493da49dc464e528ff57f28dc05bf37 [id=d24331d84ad04768ab54058120b7e72f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.071803 12772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ee96e60575425eb3e98ae1ab461180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.074416 13290 tablet_service.cc:1467] Processing CreateTablet for tablet ee5e80d295c94a24854747e468d54de6 (DEFAULT_TABLE table=default.Table_8b5a278156ad4137bf0ea955a9bb88cb [id=c4b2684e4dd74e0f9170cffcd54c2cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.077004 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5e80d295c94a24854747e468d54de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.078001 13291 tablet_service.cc:1467] Processing CreateTablet for tablet b3ee96e60575425eb3e98ae1ab461180 (DEFAULT_TABLE table=default.Table_e493da49dc464e528ff57f28dc05bf37 [id=d24331d84ad04768ab54058120b7e72f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.078318 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_1fe26210208643b1b8fb0c8da8d532d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:24.080397 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fe26210208643b1b8fb0c8da8d532d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:24.085345 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ee96e60575425eb3e98ae1ab461180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.096134 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_5e599a2567614c60aaa65d52abec3e57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:24.098294 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e599a2567614c60aaa65d52abec3e57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:24.106609 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_25e50cf2bf5742bdad77d884d8e3aa4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:24.109100 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25e50cf2bf5742bdad77d884d8e3aa4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:24.134402 12234 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e493da49dc464e528ff57f28dc05bf37" } new_table_name: "default.Table_347fb1079aa14a388ef92e410045a222" modify_external_catalogs: true
I20260810 23:32:24.161304 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 3eacc5cbe3a844779a37ca3b228f5a18 (DEFAULT_TABLE table=default.Table_1fe26210208643b1b8fb0c8da8d532d5 [id=4084badfefdb4b1fac95dcaf44590a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.163203 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eacc5cbe3a844779a37ca3b228f5a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.170063 13026 tablet_service.cc:1467] Processing CreateTablet for tablet 3eacc5cbe3a844779a37ca3b228f5a18 (DEFAULT_TABLE table=default.Table_1fe26210208643b1b8fb0c8da8d532d5 [id=4084badfefdb4b1fac95dcaf44590a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.171649 13026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eacc5cbe3a844779a37ca3b228f5a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.173421 13503 tablet_bootstrap.cc:492] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:24.186194 13503 tablet_bootstrap.cc:654] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.189150 13503 log.cc:824] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:24.193974 12229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_8b5a278156ad4137bf0ea955a9bb88cb" } modify_external_catalogs: true
I20260810 23:32:24.194295 12236 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_1fe26210208643b1b8fb0c8da8d532d5" } new_table_name: "default.Table_71caa27fb6544661a7ce026160716bb7" modify_external_catalogs: true
I20260810 23:32:24.194813 12229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_8b5a278156ad4137bf0ea955a9bb88cb" } modify_external_catalogs: true
I20260810 23:32:24.197443 12228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_4ba2f23aa6fb42398071dfe0239679f1" } modify_external_catalogs: true
I20260810 23:32:24.198161 12228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_4ba2f23aa6fb42398071dfe0239679f1" } modify_external_catalogs: true
I20260810 23:32:24.203894 13503 tablet_bootstrap.cc:492] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:24.205281 13503 ts_tablet_manager.cc:1397] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.009s
I20260810 23:32:24.256527 13509 tablet_bootstrap.cc:492] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:24.255100 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 3eacc5cbe3a844779a37ca3b228f5a18 (DEFAULT_TABLE table=default.Table_1fe26210208643b1b8fb0c8da8d532d5 [id=4084badfefdb4b1fac95dcaf44590a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.258158 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eacc5cbe3a844779a37ca3b228f5a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.258333 13503 raft_consensus.cc:348] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.259536 13503 raft_consensus.cc:374] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.259892 13503 raft_consensus.cc:729] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:24.261233 13503 consensus_queue.cc:260] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.267771 13509 tablet_bootstrap.cc:654] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.271402 13509 log.cc:824] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:24.274246 13503 ts_tablet_manager.cc:1428] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.069s user 0.048s sys 0.008s
I20260810 23:32:24.282709 13409 heartbeater.cc:503] Master 127.11.226.190:43533 was elected leader, sending a full tablet report...
I20260810 23:32:24.286139 13503 tablet_bootstrap.cc:492] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:24.302754 13503 tablet_bootstrap.cc:654] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.307915 13509 tablet_bootstrap.cc:492] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:24.309410 13509 ts_tablet_manager.cc:1397] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.056s user 0.017s sys 0.025s
I20260810 23:32:24.326382 13503 tablet_bootstrap.cc:492] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:24.326647 13514 tablet_bootstrap.cc:492] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:24.326985 13503 ts_tablet_manager.cc:1397] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.006s
W20260810 23:32:24.327814 13416 tablet.cc:2369] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:24.330565 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.189:42101
--webserver_interface=127.11.226.189
--webserver_port=37775
--builtin_ntp_servers=127.11.226.148:41081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260810 23:32:24.330511 13503 raft_consensus.cc:348] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:24.334534 13503 raft_consensus.cc:374] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.334899 13503 raft_consensus.cc:729] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:24.335758 13503 consensus_queue.cc:260] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:24.338188 13514 tablet_bootstrap.cc:654] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.338500 13503 ts_tablet_manager.cc:1428] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.011s user 0.003s sys 0.004s
I20260810 23:32:24.339803 13503 tablet_bootstrap.cc:492] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:24.343360 13514 log.cc:824] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:24.349071 13503 tablet_bootstrap.cc:654] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.352279 13509 raft_consensus.cc:348] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.353865 13509 raft_consensus.cc:374] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.354384 13509 raft_consensus.cc:729] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:24.355998 13509 consensus_queue.cc:260] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:24.362774 12236 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3eacc5cbe3a844779a37ca3b228f5a18 consensus state has no leader
I20260810 23:32:24.366834 12769 tablet_service.cc:1467] Processing CreateTablet for tablet df746466f2c9456d8261adb266a27aa8 (DEFAULT_TABLE table=default.Table_4ba2f23aa6fb42398071dfe0239679f1 [id=48e13d1cc9d2459d96a57948f250d0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.368294 12237 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet ee5e80d295c94a24854747e468d54de6 (table default.Table_8b5a278156ad4137bf0ea955a9bb88cb [id=c4b2684e4dd74e0f9170cffcd54c2cb9]). Expected version 1 got 0
I20260810 23:32:24.369110 12237 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet b3ee96e60575425eb3e98ae1ab461180 (table default.Table_e493da49dc464e528ff57f28dc05bf37 [id=d24331d84ad04768ab54058120b7e72f]). Expected version 1 got 0
I20260810 23:32:24.370733 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df746466f2c9456d8261adb266a27aa8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:24.373562 12234 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3ee96e60575425eb3e98ae1ab461180 consensus state has no leader
I20260810 23:32:24.374398 13026 tablet_service.cc:1467] Processing CreateTablet for tablet df746466f2c9456d8261adb266a27aa8 (DEFAULT_TABLE table=default.Table_4ba2f23aa6fb42398071dfe0239679f1 [id=48e13d1cc9d2459d96a57948f250d0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.376197 13026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df746466f2c9456d8261adb266a27aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.387508 13514 tablet_bootstrap.cc:492] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:24.389475 13514 ts_tablet_manager.cc:1397] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.063s user 0.018s sys 0.010s
I20260810 23:32:24.390898 13291 tablet_service.cc:1467] Processing CreateTablet for tablet df746466f2c9456d8261adb266a27aa8 (DEFAULT_TABLE table=default.Table_4ba2f23aa6fb42398071dfe0239679f1 [id=48e13d1cc9d2459d96a57948f250d0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.392215 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df746466f2c9456d8261adb266a27aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.393070 13503 tablet_bootstrap.cc:492] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:24.394059 13503 ts_tablet_manager.cc:1397] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.009s
I20260810 23:32:24.396951 13503 raft_consensus.cc:348] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.398092 13503 raft_consensus.cc:374] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.398443 13503 raft_consensus.cc:729] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:24.407213 13503 consensus_queue.cc:260] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.410431 13146 heartbeater.cc:503] Master 127.11.226.190:43533 was elected leader, sending a full tablet report...
I20260810 23:32:24.412768 12879 heartbeater.cc:503] Master 127.11.226.190:43533 was elected leader, sending a full tablet report...
I20260810 23:32:24.415438 13509 ts_tablet_manager.cc:1428] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.106s user 0.046s sys 0.027s
I20260810 23:32:24.416817 13509 tablet_bootstrap.cc:492] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:24.423995 13509 tablet_bootstrap.cc:654] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.432549 13503 ts_tablet_manager.cc:1428] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.038s user 0.007s sys 0.003s
I20260810 23:32:24.433837 13503 tablet_bootstrap.cc:492] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:24.436959 13509 tablet_bootstrap.cc:492] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:24.437552 13509 ts_tablet_manager.cc:1397] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20260810 23:32:24.440737 13509 raft_consensus.cc:348] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:24.441505 13509 raft_consensus.cc:374] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.441767 13509 raft_consensus.cc:729] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
W20260810 23:32:24.442708 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 65488.8us
I20260810 23:32:24.442741 13509 consensus_queue.cc:260] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:24.445384 13509 ts_tablet_manager.cc:1428] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 23:32:24.446367 13509 tablet_bootstrap.cc:492] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:24.453708 13509 tablet_bootstrap.cc:654] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.456439 13503 tablet_bootstrap.cc:654] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.460422 12228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet df746466f2c9456d8261adb266a27aa8
I20260810 23:32:24.463522 12768 tablet_service.cc:1514] Processing DeleteTablet for tablet df746466f2c9456d8261adb266a27aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:24.471979 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet df746466f2c9456d8261adb266a27aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:24.476434 13509 tablet_bootstrap.cc:492] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:24.476960 13509 ts_tablet_manager.cc:1397] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.004s
I20260810 23:32:24.479884 13509 raft_consensus.cc:348] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.480779 13509 raft_consensus.cc:374] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.481094 13509 raft_consensus.cc:729] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:24.481866 13509 consensus_queue.cc:260] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.475206 13514 raft_consensus.cc:348] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.489251 13509 ts_tablet_manager.cc:1428] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260810 23:32:24.490347 13509 tablet_bootstrap.cc:492] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
W20260810 23:32:24.491475 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet df746466f2c9456d8261adb266a27aa8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df746466f2c9456d8261adb266a27aa8 already in progress: creating tablet
I20260810 23:32:24.495118 13518 raft_consensus.cc:482] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:24.495751 13518 raft_consensus.cc:504] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.496300 13026 tablet_service.cc:1514] Processing DeleteTablet for tablet df746466f2c9456d8261adb266a27aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:24.502976 13518 leader_election.cc:288] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
W20260810 23:32:24.510421 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet df746466f2c9456d8261adb266a27aa8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df746466f2c9456d8261adb266a27aa8 already in progress: creating tablet
I20260810 23:32:24.483016 13514 raft_consensus.cc:374] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.512756 13514 raft_consensus.cc:729] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:24.515543 13514 consensus_queue.cc:260] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:24.518577 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet df746466f2c9456d8261adb266a27aa8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df746466f2c9456d8261adb266a27aa8 already in progress: creating tablet
I20260810 23:32:24.518963 13509 tablet_bootstrap.cc:654] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.522765 13503 tablet_bootstrap.cc:492] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:24.523347 13503 ts_tablet_manager.cc:1397] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.090s user 0.011s sys 0.020s
I20260810 23:32:24.526995 13503 raft_consensus.cc:348] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.527938 13503 raft_consensus.cc:374] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.528301 13503 raft_consensus.cc:729] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:24.532896 13503 consensus_queue.cc:260] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.535339 13503 ts_tablet_manager.cc:1428] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
W20260810 23:32:24.537155 12958 leader_election.cc:341] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: ee5e80d295c94a24854747e468d54de6
I20260810 23:32:24.545192 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_119297b47ccd4fa4afeb46b164dff00b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:24.545482 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_8d417224727445fdb1dc7c3df578e3bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:24.545781 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_4a63f0c5042c4ca7a787f8ecb6eaea3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:24.547370 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d417224727445fdb1dc7c3df578e3bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:24.547550 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a63f0c5042c4ca7a787f8ecb6eaea3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:24.548245 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_bf3c79e609d24e7cb76242c93decb907"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:24.547399 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_119297b47ccd4fa4afeb46b164dff00b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:24.552067 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf3c79e609d24e7cb76242c93decb907 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:24.556659 13514 ts_tablet_manager.cc:1428] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.167s user 0.040s sys 0.019s
I20260810 23:32:24.557838 13514 tablet_bootstrap.cc:492] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:32:24.566058 13151 tablet.cc:2369] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:24.574743 13514 tablet_bootstrap.cc:654] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.583351 12236 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 3eacc5cbe3a844779a37ca3b228f5a18 (table default.Table_71caa27fb6544661a7ce026160716bb7 [id=4084badfefdb4b1fac95dcaf44590a7f]). Expected version 1 got 0
I20260810 23:32:24.584249 12236 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet b3ee96e60575425eb3e98ae1ab461180 (table default.Table_347fb1079aa14a388ef92e410045a222 [id=d24331d84ad04768ab54058120b7e72f]). Expected version 2 got 0
I20260810 23:32:24.589174 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet b3ee96e60575425eb3e98ae1ab461180 (table default.Table_347fb1079aa14a388ef92e410045a222 [id=d24331d84ad04768ab54058120b7e72f]). Expected version 2 got 0
I20260810 23:32:24.589860 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 3eacc5cbe3a844779a37ca3b228f5a18 (table default.Table_71caa27fb6544661a7ce026160716bb7 [id=4084badfefdb4b1fac95dcaf44590a7f]). Expected version 1 got 0
I20260810 23:32:24.591001 12229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet ee5e80d295c94a24854747e468d54de6
I20260810 23:32:24.596159 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5e80d295c94a24854747e468d54de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:24.598659 12772 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5e80d295c94a24854747e468d54de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:24.602372 13026 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5e80d295c94a24854747e468d54de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:24.603502 13533 tablet_replica.cc:330] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:24.604605 13533 raft_consensus.cc:2229] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:24.605301 13533 raft_consensus.cc:2258] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:24.605350 13518 raft_consensus.cc:482] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:24.605859 13518 raft_consensus.cc:504] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.608095 13518 leader_election.cc:288] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
W20260810 23:32:24.609002 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet ee5e80d295c94a24854747e468d54de6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee5e80d295c94a24854747e468d54de6 already in progress: creating tablet
I20260810 23:32:24.609575 13533 ts_tablet_manager.cc:1905] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:24.609180 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eacc5cbe3a844779a37ca3b228f5a18" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:24.610410 12806 raft_consensus.cc:2454] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:24.611531 12772 tablet_service.cc:1467] Processing CreateTablet for tablet 75ead79616d047f9972611623fc943b1 (DEFAULT_TABLE table=default.Table_5e599a2567614c60aaa65d52abec3e57 [id=ad699de357934075aee8d7152f8e130b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.613075 12772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ead79616d047f9972611623fc943b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.613435 12958 leader_election.cc:302] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:24.614382 13518 raft_consensus.cc:2781] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:24.614730 13518 raft_consensus.cc:482] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:24.615062 13518 raft_consensus.cc:3037] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:24.620428 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 3eacc5cbe3a844779a37ca3b228f5a18 (table default.Table_71caa27fb6544661a7ce026160716bb7 [id=4084badfefdb4b1fac95dcaf44590a7f]). Expected version 1 got 0
I20260810 23:32:24.629694 13514 tablet_bootstrap.cc:492] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:24.630236 13514 ts_tablet_manager.cc:1397] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.018s
I20260810 23:32:24.637333 13532 tablet_replica.cc:330] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:24.638602 13532 raft_consensus.cc:2229] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:24.639269 13532 raft_consensus.cc:2258] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:24.641592 12226 master_service.cc:946] ReplaceTablet: received request to replace tablet 3eacc5cbe3a844779a37ca3b228f5a18 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:24.642449 12234 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet b3ee96e60575425eb3e98ae1ab461180 (table default.Table_347fb1079aa14a388ef92e410045a222 [id=d24331d84ad04768ab54058120b7e72f]). Expected version 2 got 0
I20260810 23:32:24.645049 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 75ead79616d047f9972611623fc943b1 (DEFAULT_TABLE table=default.Table_5e599a2567614c60aaa65d52abec3e57 [id=ad699de357934075aee8d7152f8e130b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.646610 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ead79616d047f9972611623fc943b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.649115 13026 tablet_service.cc:1467] Processing CreateTablet for tablet 75ead79616d047f9972611623fc943b1 (DEFAULT_TABLE table=default.Table_5e599a2567614c60aaa65d52abec3e57 [id=ad699de357934075aee8d7152f8e130b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.650379 13514 raft_consensus.cc:348] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:24.651237 13514 raft_consensus.cc:374] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.651532 13514 raft_consensus.cc:729] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:24.652280 13514 consensus_queue.cc:260] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:24.653319 12770 tablet_service.cc:1467] Processing CreateTablet for tablet af05dca78b474e8189fecae9f32a711a (DEFAULT_TABLE table=default.Table_25e50cf2bf5742bdad77d884d8e3aa4f [id=49a3a44dd7f9473d9c025d8690a00c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.653270 13031 tablet_service.cc:1467] Processing CreateTablet for tablet af05dca78b474e8189fecae9f32a711a (DEFAULT_TABLE table=default.Table_25e50cf2bf5742bdad77d884d8e3aa4f [id=49a3a44dd7f9473d9c025d8690a00c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.653820 12237 master_service.cc:946] ReplaceTablet: received request to replace tablet ee5e80d295c94a24854747e468d54de6 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:24.654944 13514 ts_tablet_manager.cc:1428] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260810 23:32:24.659567 13290 tablet_service.cc:1467] Processing CreateTablet for tablet af05dca78b474e8189fecae9f32a711a (DEFAULT_TABLE table=default.Table_25e50cf2bf5742bdad77d884d8e3aa4f [id=49a3a44dd7f9473d9c025d8690a00c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.660873 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af05dca78b474e8189fecae9f32a711a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.666669 13532 ts_tablet_manager.cc:1905] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:24.670375 13509 tablet_bootstrap.cc:492] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:24.671063 13509 ts_tablet_manager.cc:1397] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.181s user 0.022s sys 0.046s
I20260810 23:32:24.672312 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af05dca78b474e8189fecae9f32a711a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.673974 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5e80d295c94a24854747e468d54de6" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:24.676247 12952 leader_election.cc:341] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: Tablet not RUNNING: STOPPED
I20260810 23:32:24.676818 12952 leader_election.cc:302] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:24.674171 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eacc5cbe3a844779a37ca3b228f5a18" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:24.678124 13336 raft_consensus.cc:2454] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
W20260810 23:32:24.679566 12952 raft_consensus.cc:2670] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260810 23:32:24.678876 12234 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 3eacc5cbe3a844779a37ca3b228f5a18 (table default.Table_71caa27fb6544661a7ce026160716bb7 [id=4084badfefdb4b1fac95dcaf44590a7f]). Expected version 1 got 0
I20260810 23:32:24.686426 13503 tablet_bootstrap.cc:492] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:24.689759 13509 raft_consensus.cc:348] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.690721 13509 raft_consensus.cc:374] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.691066 13509 raft_consensus.cc:729] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:24.695705 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet b3ee96e60575425eb3e98ae1ab461180 (table default.Table_347fb1079aa14a388ef92e410045a222 [id=d24331d84ad04768ab54058120b7e72f]). Expected version 2 got 0
I20260810 23:32:24.696285 13503 tablet_bootstrap.cc:654] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.705204 13026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ead79616d047f9972611623fc943b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.706655 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet b3ee96e60575425eb3e98ae1ab461180 (table default.Table_347fb1079aa14a388ef92e410045a222 [id=d24331d84ad04768ab54058120b7e72f]). Expected version 2 got 0
I20260810 23:32:24.708390 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af05dca78b474e8189fecae9f32a711a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.720227 13518 raft_consensus.cc:504] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.723762 13518 leader_election.cc:288] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:24.725958 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eacc5cbe3a844779a37ca3b228f5a18" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:24.726675 12806 raft_consensus.cc:3037] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:24.727855 12226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 3eacc5cbe3a844779a37ca3b228f5a18
I20260810 23:32:24.729820 12226 catalog_manager.cc:6565] ReplaceTablet: tablet 3eacc5cbe3a844779a37ca3b228f5a18 deleted and replaced by tablet ebdfad02b9b64ed4a1f781bc44f4c4e4
I20260810 23:32:24.731679 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eacc5cbe3a844779a37ca3b228f5a18 with delete_type TABLET_DATA_DELETED (replaced by tablet ebdfad02b9b64ed4a1f781bc44f4c4e4) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:24.734552 12768 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eacc5cbe3a844779a37ca3b228f5a18 with delete_type TABLET_DATA_DELETED (replaced by tablet ebdfad02b9b64ed4a1f781bc44f4c4e4) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:24.735678 13526 tablet_replica.cc:330] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:24.735817 12226 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_25e50cf2bf5742bdad77d884d8e3aa4f" } new_table_name: "default.Table_a4956dacaf2146caa115cf54303141eb" modify_external_catalogs: true
I20260810 23:32:24.691906 13509 consensus_queue.cc:260] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.742003 12240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_71caa27fb6544661a7ce026160716bb7" } modify_external_catalogs: true
I20260810 23:32:24.742594 12240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_71caa27fb6544661a7ce026160716bb7" } modify_external_catalogs: true
I20260810 23:32:24.743295 13509 ts_tablet_manager.cc:1428] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.072s user 0.003s sys 0.003s
I20260810 23:32:24.655907 13514 tablet_bootstrap.cc:492] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:32:24.757189 12891 tablet.cc:2369] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:24.764954 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eacc5cbe3a844779a37ca3b228f5a18" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:24.765667 13336 raft_consensus.cc:3037] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:24.767467 12226 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af05dca78b474e8189fecae9f32a711a consensus state has no leader
I20260810 23:32:24.769415 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet df746466f2c9456d8261adb266a27aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:24.771700 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eacc5cbe3a844779a37ca3b228f5a18 with delete_type TABLET_DATA_DELETED (replaced by tablet ebdfad02b9b64ed4a1f781bc44f4c4e4) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:24.796772 13533 ts_tablet_manager.cc:1918] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:24.797636 13533 log.cc:1196] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/ee5e80d295c94a24854747e468d54de6
I20260810 23:32:24.799357 13533 ts_tablet_manager.cc:1939] T ee5e80d295c94a24854747e468d54de6 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:24.801733 12806 raft_consensus.cc:2454] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:24.804140 13526 raft_consensus.cc:2229] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:24.803825 12958 leader_election.cc:302] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:24.806630 13518 raft_consensus.cc:2781] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:24.813814 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 151ba1dd24574739a9f0d33c2257235d (DEFAULT_TABLE table=default.Table_4a63f0c5042c4ca7a787f8ecb6eaea3d [id=32144cd75df84cfabce64657802c111a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.813870 12226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_8d417224727445fdb1dc7c3df578e3bc" } modify_external_catalogs: true
I20260810 23:32:24.814983 12226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_8d417224727445fdb1dc7c3df578e3bc" } modify_external_catalogs: true
I20260810 23:32:24.817193 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151ba1dd24574739a9f0d33c2257235d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.795429 12237 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet b3ee96e60575425eb3e98ae1ab461180 (table default.Table_347fb1079aa14a388ef92e410045a222 [id=d24331d84ad04768ab54058120b7e72f]). Expected version 2 got 0
I20260810 23:32:24.826853 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_edb5813389c6488ea72bae6a8373f405"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:24.830773 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edb5813389c6488ea72bae6a8373f405 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:24.831436 12234 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_347fb1079aa14a388ef92e410045a222" } new_table_name: "default.Table_e987219e3104474daa72d065991a72dd" modify_external_catalogs: true
I20260810 23:32:24.842121 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 151ba1dd24574739a9f0d33c2257235d (DEFAULT_TABLE table=default.Table_4a63f0c5042c4ca7a787f8ecb6eaea3d [id=32144cd75df84cfabce64657802c111a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.844734 13025 tablet_service.cc:1467] Processing CreateTablet for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f (DEFAULT_TABLE table=default.Table_8d417224727445fdb1dc7c3df578e3bc [id=d1a1db95f5f84899b7da1b07aa3d5083]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.846843 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_e72ecc5585cb44cea2bbe36fb20b3a6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:24.848084 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.849534 13518 raft_consensus.cc:686] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
W20260810 23:32:24.850107 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e72ecc5585cb44cea2bbe36fb20b3a6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:24.851307 13518 consensus_queue.cc:237] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.856920 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151ba1dd24574739a9f0d33c2257235d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.838115 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 151ba1dd24574739a9f0d33c2257235d (DEFAULT_TABLE table=default.Table_4a63f0c5042c4ca7a787f8ecb6eaea3d [id=32144cd75df84cfabce64657802c111a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.882002 13509 tablet_bootstrap.cc:492] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:24.888937 13503 tablet_bootstrap.cc:492] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:24.890111 13503 ts_tablet_manager.cc:1397] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.204s user 0.011s sys 0.006s
I20260810 23:32:24.905812 13336 raft_consensus.cc:2454] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:24.905401 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f (DEFAULT_TABLE table=default.Table_8d417224727445fdb1dc7c3df578e3bc [id=d1a1db95f5f84899b7da1b07aa3d5083]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.909705 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.918105 13503 raft_consensus.cc:348] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.920264 13503 raft_consensus.cc:374] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.921283 13503 raft_consensus.cc:729] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:24.926512 13503 consensus_queue.cc:260] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:24.940502 13552 raft_consensus.cc:482] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:24.942351 13552 raft_consensus.cc:504] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:24.944046 12229 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_90dfb8b49e7d41fc8267c506f1e3f57e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:24.948132 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_5819e8073eb6404886ae221a7bc63155"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:24.968804 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet ee5e80d295c94a24854747e468d54de6 (table default.Table_8b5a278156ad4137bf0ea955a9bb88cb [id=c4b2684e4dd74e0f9170cffcd54c2cb9]) successfully deleted
W20260810 23:32:24.970883 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 3eacc5cbe3a844779a37ca3b228f5a18 (table default.Table_71caa27fb6544661a7ce026160716bb7 [id=4084badfefdb4b1fac95dcaf44590a7f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3eacc5cbe3a844779a37ca3b228f5a18 consensus state has no leader
I20260810 23:32:24.974141 13511 raft_consensus.cc:482] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:24.975914 13511 raft_consensus.cc:504] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:24.863965 13526 raft_consensus.cc:2258] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:24.984076 12223 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_4a63f0c5042c4ca7a787f8ecb6eaea3d" } new_table_name: "default.Table_0af7ca97e51c47a29d5dd8867b152066" modify_external_catalogs: true
I20260810 23:32:24.993389 13533 tablet_replica.cc:330] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
W20260810 23:32:24.997669 12229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90dfb8b49e7d41fc8267c506f1e3f57e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:24.997145 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f (DEFAULT_TABLE table=default.Table_8d417224727445fdb1dc7c3df578e3bc [id=d1a1db95f5f84899b7da1b07aa3d5083]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.874713 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151ba1dd24574739a9f0d33c2257235d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:24.998828 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5819e8073eb6404886ae221a7bc63155 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:25.042505 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ee96e60575425eb3e98ae1ab461180" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:25.055719 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:25.063783 13533 raft_consensus.cc:2229] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:25.063719 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ee96e60575425eb3e98ae1ab461180" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:25.065011 13336 raft_consensus.cc:2454] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:25.067109 12952 leader_election.cc:302] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:25.084308 12806 raft_consensus.cc:2454] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:25.089234 13532 ts_tablet_manager.cc:1918] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:25.093993 12228 master_service.cc:946] ReplaceTablet: received request to replace tablet af05dca78b474e8189fecae9f32a711a from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:25.072423 13552 leader_election.cc:288] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:25.078500 13554 raft_consensus.cc:2781] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:25.102628 13554 raft_consensus.cc:482] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:25.103533 13554 raft_consensus.cc:3037] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:25.071306 13509 tablet_bootstrap.cc:654] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:25.107286 12240 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet ebdfad02b9b64ed4a1f781bc44f4c4e4
I20260810 23:32:25.120877 13532 log.cc:1196] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/ee5e80d295c94a24854747e468d54de6
I20260810 23:32:25.121403 13526 ts_tablet_manager.cc:1905] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:25.123128 13532 ts_tablet_manager.cc:1939] T ee5e80d295c94a24854747e468d54de6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:25.125247 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5e80d295c94a24854747e468d54de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:25.118886 13503 ts_tablet_manager.cc:1428] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.228s user 0.013s sys 0.013s
I20260810 23:32:25.141299 13533 pending_rounds.cc:62] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b: Trying to abort 1 pending ops.
I20260810 23:32:25.142851 13533 pending_rounds.cc:68] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317113835115245568 op_type: NO_OP noop_request { }
I20260810 23:32:25.144572 13503 tablet_bootstrap.cc:492] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:25.145735 13533 raft_consensus.cc:2866] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:32:25.151643 13533 raft_consensus.cc:2258] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:25.154752 13555 raft_consensus.cc:482] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:25.156960 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet df746466f2c9456d8261adb266a27aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:25.179939 13511 leader_election.cc:288] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:25.192791 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet ee5e80d295c94a24854747e468d54de6 (table default.Table_8b5a278156ad4137bf0ea955a9bb88cb [id=c4b2684e4dd74e0f9170cffcd54c2cb9]) successfully deleted
I20260810 23:32:25.155620 13555 raft_consensus.cc:504] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:25.203780 13509 tablet_bootstrap.cc:492] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:25.205051 13509 ts_tablet_manager.cc:1397] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.324s user 0.014s sys 0.042s
W20260810 23:32:25.202456 12772 consensus_meta.cc:287] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.501s user 0.009s sys 0.007s
I20260810 23:32:25.220443 13555 leader_election.cc:288] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:25.221145 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_572f84e499024608b6d500c2979d411c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:25.232409 13554 raft_consensus.cc:504] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:25.235520 13503 tablet_bootstrap.cc:654] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:25.223800 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_572f84e499024608b6d500c2979d411c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:25.210361 13514 tablet_bootstrap.cc:700] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent opening tablet: real 0.135s user 0.004s sys 0.001s
I20260810 23:32:25.235841 12226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 7be44f795bcd4cdcbdb2f3a03ce5253f
I20260810 23:32:25.238356 13514 tablet_bootstrap.cc:654] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:25.252594 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 77009.4us
I20260810 23:32:25.264370 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:25.241243 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:25.253997 13532 tablet_replica.cc:330] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:25.269343 13532 raft_consensus.cc:2229] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:25.270287 13532 raft_consensus.cc:2258] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:25.271206 13533 ts_tablet_manager.cc:1905] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:25.275178 13509 raft_consensus.cc:348] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:25.280889 13509 raft_consensus.cc:374] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:25.281463 13509 raft_consensus.cc:729] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:25.281936 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ee96e60575425eb3e98ae1ab461180" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:25.282761 13509 consensus_queue.cc:260] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:25.297049 13532 ts_tablet_manager.cc:1905] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:25.283164 13336 raft_consensus.cc:3037] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:25.279340 13554 leader_election.cc:288] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:25.306054 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ee96e60575425eb3e98ae1ab461180" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:25.307744 12806 raft_consensus.cc:3037] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:25.318537 12772 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:25.321033 12235 master_service.cc:946] ReplaceTablet: received request to replace tablet b3ee96e60575425eb3e98ae1ab461180 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:25.321444 13509 ts_tablet_manager.cc:1428] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.116s user 0.015s sys 0.016s
I20260810 23:32:25.329142 13509 tablet_bootstrap.cc:492] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:25.323868 12237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_119297b47ccd4fa4afeb46b164dff00b" } modify_external_catalogs: true
I20260810 23:32:25.333806 12237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_119297b47ccd4fa4afeb46b164dff00b" } modify_external_catalogs: true
I20260810 23:32:25.390997 13552 raft_consensus.cc:482] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:25.392716 13552 raft_consensus.cc:504] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:25.401571 13552 leader_election.cc:288] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:25.405655 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df746466f2c9456d8261adb266a27aa8" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:25.409848 12952 leader_election.cc:341] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: Tablet not RUNNING: STOPPED
I20260810 23:32:25.428315 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df746466f2c9456d8261adb266a27aa8" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:25.434239 13509 tablet_bootstrap.cc:654] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:25.439968 12234 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3ee96e60575425eb3e98ae1ab461180 consensus state has no leader
I20260810 23:32:25.454874 13503 tablet_bootstrap.cc:492] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
W20260810 23:32:25.457726 12958 leader_election.cc:341] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:25.458662 12958 leader_election.cc:302] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:25.461621 13552 raft_consensus.cc:2726] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:25.471213 13503 ts_tablet_manager.cc:1397] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.335s user 0.019s sys 0.016s
I20260810 23:32:25.455276 12234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e72ecc5585cb44cea2bbe36fb20b3a6c" } modify_external_catalogs: true
I20260810 23:32:25.481892 13509 tablet_bootstrap.cc:492] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:25.483319 13509 ts_tablet_manager.cc:1397] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.155s user 0.016s sys 0.039s
I20260810 23:32:25.483530 12234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e72ecc5585cb44cea2bbe36fb20b3a6c" } modify_external_catalogs: true
I20260810 23:32:25.487414 13336 raft_consensus.cc:2454] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:25.490654 12952 leader_election.cc:302] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:25.493214 13552 raft_consensus.cc:2781] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:25.494506 13552 raft_consensus.cc:686] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:25.497208 13552 consensus_queue.cc:237] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:25.505898 13509 raft_consensus.cc:348] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:25.511030 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df746466f2c9456d8261adb266a27aa8" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:25.514186 13509 raft_consensus.cc:374] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:25.515612 13509 raft_consensus.cc:729] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:25.519976 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ee96e60575425eb3e98ae1ab461180" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
W20260810 23:32:25.521286 13213 leader_election.cc:341] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:25.526878 13509 consensus_queue.cc:260] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:25.548933 13509 ts_tablet_manager.cc:1428] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.064s user 0.012s sys 0.015s
I20260810 23:32:25.554368 13509 tablet_bootstrap.cc:492] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:25.562525 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df746466f2c9456d8261adb266a27aa8" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:25.595399 12229 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_10156b698d054ae8845a67255a9daf3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:25.575079 13532 ts_tablet_manager.cc:1918] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:25.605000 13532 log.cc:1196] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/df746466f2c9456d8261adb266a27aa8
W20260810 23:32:25.606383 12229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10156b698d054ae8845a67255a9daf3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:25.606825 13532 ts_tablet_manager.cc:1939] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:25.568557 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ee96e60575425eb3e98ae1ab461180" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:25.618386 13514 tablet_bootstrap.cc:492] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:25.611086 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet df746466f2c9456d8261adb266a27aa8 (table default.Table_4ba2f23aa6fb42398071dfe0239679f1 [id=48e13d1cc9d2459d96a57948f250d0c8]) successfully deleted
I20260810 23:32:25.577512 13060 raft_consensus.cc:2454] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:25.634338 13212 leader_election.cc:302] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
W20260810 23:32:25.636767 12223 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 151ba1dd24574739a9f0d33c2257235d consensus state has no leader
I20260810 23:32:25.638542 13555 raft_consensus.cc:2726] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:25.643194 13212 leader_election.cc:302] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
W20260810 23:32:25.650189 13212 raft_consensus.cc:2670] T df746466f2c9456d8261adb266a27aa8 P f54aa78a60784a3bb2f5eb4bb2401ce3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260810 23:32:25.653811 13503 raft_consensus.cc:348] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:25.656186 13503 raft_consensus.cc:374] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:25.657055 13503 raft_consensus.cc:729] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:25.666344 12806 raft_consensus.cc:2454] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:25.665987 12223 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_5e599a2567614c60aaa65d52abec3e57" } new_table_name: "default.Table_e0603d4692934dce9e07597811ccad0a" modify_external_catalogs: true
I20260810 23:32:25.669437 13532 tablet_replica.cc:330] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:25.672521 13532 raft_consensus.cc:2229] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:25.674306 13532 raft_consensus.cc:2258] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:25.675315 13514 ts_tablet_manager.cc:1397] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 1.020s user 0.028s sys 0.009s
I20260810 23:32:25.659459 13503 consensus_queue.cc:260] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:25.733325 13532 ts_tablet_manager.cc:1905] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:25.738520 13574 log.cc:924] Time spent T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.054s user 0.004s sys 0.000s
I20260810 23:32:25.740376 13555 raft_consensus.cc:482] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:25.741696 13555 raft_consensus.cc:504] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:25.748898 13555 leader_election.cc:288] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:25.753082 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ead79616d047f9972611623fc943b1" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
W20260810 23:32:25.756814 13213 leader_election.cc:341] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:25.764868 12228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet af05dca78b474e8189fecae9f32a711a
I20260810 23:32:25.769104 13552 raft_consensus.cc:482] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:25.770258 13552 raft_consensus.cc:504] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:25.770418 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ead79616d047f9972611623fc943b1" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:25.775668 13552 leader_election.cc:288] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:25.777387 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af05dca78b474e8189fecae9f32a711a" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:25.778343 13060 raft_consensus.cc:2454] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:25.779012 13336 raft_consensus.cc:2454] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:25.785463 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af05dca78b474e8189fecae9f32a711a" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:25.788254 12952 leader_election.cc:302] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:25.788815 12772 tablet_service.cc:1514] Processing DeleteTablet for tablet af05dca78b474e8189fecae9f32a711a with delete_type TABLET_DATA_DELETED (replaced by tablet 217a7582902244f9b0ca65be1c20af22) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:25.790509 13552 raft_consensus.cc:2781] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:25.791447 13552 raft_consensus.cc:482] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:25.792876 13552 raft_consensus.cc:3037] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:25.791383 13212 leader_election.cc:302] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:25.797425 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet af05dca78b474e8189fecae9f32a711a with delete_type TABLET_DATA_DELETED (replaced by tablet 217a7582902244f9b0ca65be1c20af22) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:25.797914 13555 raft_consensus.cc:2781] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:25.799201 13555 raft_consensus.cc:482] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:25.800295 13555 raft_consensus.cc:3037] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:25.804875 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_b2d2da314c7e45c182359a84356a67f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:25.811048 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2d2da314c7e45c182359a84356a67f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:25.828166 12228 catalog_manager.cc:6565] ReplaceTablet: tablet af05dca78b474e8189fecae9f32a711a deleted and replaced by tablet 217a7582902244f9b0ca65be1c20af22
I20260810 23:32:25.840323 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet af05dca78b474e8189fecae9f32a711a with delete_type TABLET_DATA_DELETED (replaced by tablet 217a7582902244f9b0ca65be1c20af22) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:25.842626 13503 ts_tablet_manager.cc:1428] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.370s user 0.024s sys 0.007s
W20260810 23:32:25.791849 12958 leader_election.cc:341] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:25.854152 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 62455.5us
I20260810 23:32:25.879833 12228 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_a4956dacaf2146caa115cf54303141eb" } new_table_name: "default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c" modify_external_catalogs: true
I20260810 23:32:25.911854 13514 raft_consensus.cc:348] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:25.915262 13514 raft_consensus.cc:374] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:25.932516 13503 tablet_bootstrap.cc:492] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:25.949433 13514 raft_consensus.cc:729] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:25.946802 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_fd3d342652da4d5baad3608431df6041"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:25.951850 13514 consensus_queue.cc:260] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:25.953814 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd3d342652da4d5baad3608431df6041 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:25.986225 13555 raft_consensus.cc:504] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:25.991288 12235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet b3ee96e60575425eb3e98ae1ab461180
I20260810 23:32:26.000842 13509 tablet_bootstrap.cc:654] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:26.006845 12772 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ee96e60575425eb3e98ae1ab461180 with delete_type TABLET_DATA_DELETED (replaced by tablet 922ce728af9740108dc959b3155ffca7) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:26.008291 12237 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet cb33f3edbb5b462998520bd529f5fd5b
I20260810 23:32:26.008435 12235 catalog_manager.cc:6565] ReplaceTablet: tablet b3ee96e60575425eb3e98ae1ab461180 deleted and replaced by tablet 922ce728af9740108dc959b3155ffca7
I20260810 23:32:26.010684 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ee96e60575425eb3e98ae1ab461180 with delete_type TABLET_DATA_DELETED (replaced by tablet 922ce728af9740108dc959b3155ffca7) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:26.013070 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ead79616d047f9972611623fc943b1" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:26.014509 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ee96e60575425eb3e98ae1ab461180 with delete_type TABLET_DATA_DELETED (replaced by tablet 922ce728af9740108dc959b3155ffca7) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:26.016506 13213 leader_election.cc:341] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:26.043809 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ead79616d047f9972611623fc943b1" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:26.050910 13060 raft_consensus.cc:3037] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:26.053090 12768 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 10) took 1070 ms. Trace:
W20260810 23:32:26.058789 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 151ba1dd24574739a9f0d33c2257235d (table default.Table_0af7ca97e51c47a29d5dd8867b152066 [id=32144cd75df84cfabce64657802c111a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 151ba1dd24574739a9f0d33c2257235d consensus state has no leader
I20260810 23:32:26.066448 12237 master_service.cc:946] ReplaceTablet: received request to replace tablet 151ba1dd24574739a9f0d33c2257235d from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:26.025512 12770 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 9) took 1190 ms. Trace:
I20260810 23:32:26.084506 13555 leader_election.cc:288] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:26.097086 13060 raft_consensus.cc:2454] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:26.099673 13532 ts_tablet_manager.cc:1918] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:26.100481 13212 leader_election.cc:302] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:26.081593 12770 rpcz_store.cc:276] 0810 23:32:24.835845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:24.836264 (+ 419us) service_pool.cc:227] Handling call
0810 23:32:24.873076 (+ 36812us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:24.873387 (+ 311us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:25.585201 (+711814us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:26.023589 (+438388us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:26.025361 (+ 1772us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:26.105687 13555 raft_consensus.cc:2781] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:26.104645 13526 ts_tablet_manager.cc:1918] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:26.116710 13532 log.cc:1196] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/3eacc5cbe3a844779a37ca3b228f5a18
I20260810 23:32:26.116886 13526 log.cc:1196] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/3eacc5cbe3a844779a37ca3b228f5a18
I20260810 23:32:26.121483 13526 ts_tablet_manager.cc:1939] T 3eacc5cbe3a844779a37ca3b228f5a18 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:26.129145 13532 ts_tablet_manager.cc:1939] T 3eacc5cbe3a844779a37ca3b228f5a18 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:26.080539 12768 rpcz_store.cc:276] 0810 23:32:24.982663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:24.985693 (+ 3030us) service_pool.cc:227] Handling call
0810 23:32:25.037341 (+ 51648us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:25.041486 (+ 4145us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:25.603097 (+561611us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:26.051438 (+448341us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:26.052928 (+ 1490us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:26.133725 13555 raft_consensus.cc:686] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:26.144061 13555 consensus_queue.cc:237] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:26.150743 13533 ts_tablet_manager.cc:1918] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:26.161751 12234 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet a92d1b9d934744b3b268564656c42b1a
I20260810 23:32:26.151813 13533 log.cc:1196] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/3eacc5cbe3a844779a37ca3b228f5a18
I20260810 23:32:26.184912 13514 ts_tablet_manager.cc:1428] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.479s user 0.017s sys 0.015s
I20260810 23:32:26.191001 13533 ts_tablet_manager.cc:1939] T 3eacc5cbe3a844779a37ca3b228f5a18 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:26.197786 13514 tablet_bootstrap.cc:492] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:26.110853 13552 raft_consensus.cc:504] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:26.214655 13533 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 8) took 1483 ms. Trace:
I20260810 23:32:26.223920 13533 rpcz_store.cc:276] 0810 23:32:24.731286 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:24.731485 (+ 199us) service_pool.cc:227] Handling call
0810 23:32:24.992884 (+261399us) ts_tablet_manager.cc:1145] Deleting tablet 3eacc5cbe3a844779a37ca3b228f5a18
0810 23:32:24.993037 (+ 153us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:25.092411 (+ 99374us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b54000c0154. stack: 00007fa86ed56033 00007fa86ed55f30 00007fa86e8d7534 00000000004db2b4 00000000004db249 00007fa875c62471 00007fa875ce5352 00007fa8760fb26c 00007fa876d08cfa 00007fa876d38e10 00007fa876d1fe0d 00007fa876d1fdb9 00007fa876d1fd49 00007fa876d1fd11 00007fa876d1ef3d 0000000000580784
0810 23:32:25.779330 (+686919us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:26.150167 (+370837us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:26.214423 (+ 64256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49708928,"tablet-delete.queue_time_us":260469}
I20260810 23:32:26.237895 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af05dca78b474e8189fecae9f32a711a" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
W20260810 23:32:26.241456 12958 leader_election.cc:341] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:26.247697 12235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_572f84e499024608b6d500c2979d411c" } modify_external_catalogs: true
I20260810 23:32:26.247856 13533 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 11) took 1123 ms. Trace:
I20260810 23:32:26.250483 12235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_572f84e499024608b6d500c2979d411c" } modify_external_catalogs: true
I20260810 23:32:26.252986 12236 master_service.cc:946] ReplaceTablet: received request to replace tablet 7be44f795bcd4cdcbdb2f3a03ce5253f from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:26.257144 13533 rpcz_store.cc:276] 0810 23:32:25.124624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:25.124955 (+ 331us) service_pool.cc:227] Handling call
0810 23:32:26.246656 (+1121701us) ts_tablet_manager.cc:1145] Deleting tablet ee5e80d295c94a24854747e468d54de6
0810 23:32:26.246864 (+ 208us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:26.247572 (+ 708us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1119398}
W20260810 23:32:26.266570 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet ee5e80d295c94a24854747e468d54de6 because the tablet was not found. No further retry: Not found: Tablet not found: ee5e80d295c94a24854747e468d54de6
I20260810 23:32:26.266991 13526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 8) took 1532 ms. Trace:
I20260810 23:32:26.260179 13552 leader_election.cc:288] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:26.274188 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_b4921435057046b6bdca92df0d5284dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:26.277872 12228 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 217a7582902244f9b0ca65be1c20af22 has no consensus state
I20260810 23:32:26.221674 13503 tablet_bootstrap.cc:654] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:26.258011 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af05dca78b474e8189fecae9f32a711a" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:26.287592 12768 tablet_service.cc:1514] Processing DeleteTablet for tablet df746466f2c9456d8261adb266a27aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:26.292642 13336 raft_consensus.cc:3037] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:26.292886 13526 rpcz_store.cc:276] 0810 23:32:24.734121 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:24.734318 (+ 197us) service_pool.cc:227] Handling call
0810 23:32:24.735214 (+ 896us) ts_tablet_manager.cc:1145] Deleting tablet 3eacc5cbe3a844779a37ca3b228f5a18
0810 23:32:24.735285 (+ 71us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:24.863330 (+128045us) spinlock_profiling.cc:243] Waited 67.4 ms on lock 0x7b54000c08d4. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3c5352 00007f3cde7db26c 00007f3cdf3e8cfa 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49 00007f3cdf3ffd11 00007f3cdf3fef3d 0000000000580784
0810 23:32:25.709987 (+846657us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:26.103832 (+393845us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:26.266692 (+162860us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":154926336,"tablet-delete.queue_time_us":133}
I20260810 23:32:26.310859 13526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 11) took 1070 ms. Trace:
I20260810 23:32:26.314378 13532 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 9) took 1544 ms. Trace:
I20260810 23:32:26.312287 13526 rpcz_store.cc:276] 0810 23:32:25.240418 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:25.264612 (+ 24194us) service_pool.cc:227] Handling call
0810 23:32:26.307278 (+1042666us) ts_tablet_manager.cc:1145] Deleting tablet 7be44f795bcd4cdcbdb2f3a03ce5253f
0810 23:32:26.307356 (+ 78us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:26.310655 (+ 3299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":987545}
I20260810 23:32:26.329222 13509 tablet_bootstrap.cc:492] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
W20260810 23:32:26.334514 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7be44f795bcd4cdcbdb2f3a03ce5253f already in progress: creating tablet
I20260810 23:32:26.342114 13509 ts_tablet_manager.cc:1397] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.789s user 0.024s sys 0.041s
I20260810 23:32:26.327661 13533 tablet_replica.cc:330] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
W20260810 23:32:26.343401 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet af05dca78b474e8189fecae9f32a711a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af05dca78b474e8189fecae9f32a711a already in progress: creating tablet
I20260810 23:32:26.352272 13533 raft_consensus.cc:2229] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:26.358265 13533 raft_consensus.cc:2258] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:26.368654 13526 tablet_replica.cc:330] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
W20260810 23:32:26.369520 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4921435057046b6bdca92df0d5284dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:26.379875 13526 raft_consensus.cc:2229] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:26.381278 13526 raft_consensus.cc:2258] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:26.381544 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_69a8d6b34dd84459a083c452460bd345"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:26.388293 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69a8d6b34dd84459a083c452460bd345 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:26.396306 13532 rpcz_store.cc:276] 0810 23:32:24.769497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:24.770674 (+ 1177us) service_pool.cc:227] Handling call
0810 23:32:25.668846 (+898172us) ts_tablet_manager.cc:1145] Deleting tablet 3eacc5cbe3a844779a37ca3b228f5a18
0810 23:32:25.669061 (+ 215us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:25.972860 (+303799us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:26.099117 (+126257us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:26.314243 (+215126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":896183}
I20260810 23:32:26.401062 12229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_bf3c79e609d24e7cb76242c93decb907" } modify_external_catalogs: true
I20260810 23:32:26.403743 12229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_bf3c79e609d24e7cb76242c93decb907" } modify_external_catalogs: true
W20260810 23:32:26.416971 12958 consensus_peers.cc:487] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:26.423172 13576 raft_consensus.cc:482] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:26.424495 13576 raft_consensus.cc:504] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:26.431188 13533 ts_tablet_manager.cc:1905] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:26.435225 13576 leader_election.cc:288] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:26.443569 13526 ts_tablet_manager.cc:1905] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:26.452101 13509 raft_consensus.cc:348] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:26.462569 13509 raft_consensus.cc:374] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:26.464078 13509 raft_consensus.cc:729] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:26.413388 13503 tablet_bootstrap.cc:492] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:26.450697 13336 raft_consensus.cc:2454] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:26.465343 13503 ts_tablet_manager.cc:1397] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.618s user 0.006s sys 0.044s
I20260810 23:32:26.468366 12952 leader_election.cc:302] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:26.477674 13552 raft_consensus.cc:2781] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:26.484968 13509 consensus_queue.cc:260] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:26.476181 13503 raft_consensus.cc:348] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:26.497220 13503 raft_consensus.cc:374] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:26.509784 13503 raft_consensus.cc:729] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:26.520174 13503 consensus_queue.cc:260] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:26.531131 13503 ts_tablet_manager.cc:1428] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.065s user 0.013s sys 0.005s
I20260810 23:32:26.515969 13532 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 12) took 1275 ms. Trace:
I20260810 23:32:26.543691 13532 rpcz_store.cc:276] 0810 23:32:25.240807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:25.241044 (+ 237us) service_pool.cc:227] Handling call
0810 23:32:26.467522 (+1226478us) ts_tablet_manager.cc:1145] Deleting tablet 7be44f795bcd4cdcbdb2f3a03ce5253f
0810 23:32:26.467650 (+ 128us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:26.515169 (+ 47519us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1201034}
W20260810 23:32:26.548197 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7be44f795bcd4cdcbdb2f3a03ce5253f already in progress: creating tablet
I20260810 23:32:26.549360 13532 tablet_replica.cc:330] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:26.503273 13514 tablet_bootstrap.cc:654] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:26.570358 13552 raft_consensus.cc:686] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
W20260810 23:32:26.546270 13589 log.cc:924] Time spent T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Append to log took a long time: real 0.054s user 0.003s sys 0.000s
I20260810 23:32:26.611811 13509 ts_tablet_manager.cc:1428] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.269s user 0.020s sys 0.010s
I20260810 23:32:26.634452 13532 raft_consensus.cc:2229] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:26.650241 13509 tablet_bootstrap.cc:492] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:26.659082 13503 tablet_bootstrap.cc:492] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:26.680320 13532 raft_consensus.cc:2258] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:26.703948 12223 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75ead79616d047f9972611623fc943b1 consensus state has no leader
I20260810 23:32:26.703409 13600 consensus_queue.cc:1035] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:32:26.709301 12226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 8) took 1455 ms (client timeout 1000 ms). Trace:
W20260810 23:32:26.713325 12226 rpcz_store.cc:269] 0810 23:32:25.254392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:25.254782 (+ 390us) service_pool.cc:227] Handling call
0810 23:32:26.709143 (+1454361us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260810 23:32:26.645612 13552 consensus_queue.cc:237] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:26.740991 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 26b0c3f3068744e0899c7bb1e2823282 (DEFAULT_TABLE table=default.Table_bf3c79e609d24e7cb76242c93decb907 [id=afcd22d5c266477cbc0f2160f852346b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:26.741047 13025 tablet_service.cc:1467] Processing CreateTablet for tablet 26b0c3f3068744e0899c7bb1e2823282 (DEFAULT_TABLE table=default.Table_bf3c79e609d24e7cb76242c93decb907 [id=afcd22d5c266477cbc0f2160f852346b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:26.745153 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b0c3f3068744e0899c7bb1e2823282. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:26.748660 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b0c3f3068744e0899c7bb1e2823282. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:26.705513 12223 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 60) took 1046 ms. Trace:
W20260810 23:32:26.725389 12224 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 13) took 1393 ms (client timeout 1000 ms). Trace:
W20260810 23:32:26.760457 12224 rpcz_store.cc:269] 0810 23:32:25.332559 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:25.332848 (+ 289us) service_pool.cc:227] Handling call
0810 23:32:26.725230 (+1392382us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260810 23:32:26.767905 13555 raft_consensus.cc:482] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:26.769415 13555 raft_consensus.cc:504] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:26.777000 13555 leader_election.cc:288] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:26.735896 13532 ts_tablet_manager.cc:1905] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:26.780786 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151ba1dd24574739a9f0d33c2257235d" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:26.782286 13060 raft_consensus.cc:2454] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:26.692337 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_85ae9e6bdba74ff0bcb1027b39707b5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:26.786602 13212 leader_election.cc:302] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
W20260810 23:32:26.787278 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85ae9e6bdba74ff0bcb1027b39707b5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:26.789036 13555 raft_consensus.cc:2781] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:26.789885 13555 raft_consensus.cc:482] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:26.790747 13555 raft_consensus.cc:3037] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:26.813104 12215 consensus_peers.cc:487] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 -> Peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Couldn't send request to peer 198f534261994047a68e1d509bb2fd8b. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260810 23:32:26.833341 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 26b0c3f3068744e0899c7bb1e2823282 (DEFAULT_TABLE table=default.Table_bf3c79e609d24e7cb76242c93decb907 [id=afcd22d5c266477cbc0f2160f852346b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:26.781380 12226 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_5819e8073eb6404886ae221a7bc63155" } new_table_name: "default.Table_085f24edadf34deea7a3d9356d00c7b1" modify_external_catalogs: true
I20260810 23:32:26.837297 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b0c3f3068744e0899c7bb1e2823282. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:26.840430 13514 tablet_bootstrap.cc:492] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:26.841921 13514 ts_tablet_manager.cc:1397] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.654s user 0.034s sys 0.034s
I20260810 23:32:26.844162 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151ba1dd24574739a9f0d33c2257235d" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:26.856432 13604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e964330cf113403a8aa5b73ddd3b31d0)
I20260810 23:32:26.858098 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df746466f2c9456d8261adb266a27aa8" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:26.865967 12700 leader_election.cc:341] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: Tablet not RUNNING: STOPPED
I20260810 23:32:26.798108 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_aa1ea1ebbc1d4264a5c9f638092a5825"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:26.916581 13213 leader_election.cc:341] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:26.755072 12223 rpcz_store.cc:276] 0810 23:32:25.659400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:25.661067 (+ 1667us) service_pool.cc:227] Handling call
0810 23:32:25.665712 (+ 4645us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e77d6 00007f0ce6e2d634 00007f0ce6e2d5c2 00007f0ce6e2d4f1 00007f0ce6e2d46c 00007f0ce6e2c722 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:25.668254 (+ 2542us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:25.670211 (+ 1957us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:25.693197 (+ 22986us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:25.694315 (+ 1118us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:26.199516 (+505201us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:26.705344 (+505828us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:25.925036 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:25.927111 (+ 2075us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:25.927206 (+ 95us) write_op.cc:435] Acquired schema lock
0810 23:32:25.927223 (+ 17us) tablet.cc:660] Decoding operations
0810 23:32:25.928651 (+ 1428us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:25.928826 (+ 175us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:25.928901 (+ 75us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:25.929631 (+ 730us) tablet.cc:699] Row locks acquired
0810 23:32:25.929649 (+ 18us) write_op.cc:260] PREPARE: finished
0810 23:32:25.930161 (+ 512us) write_op.cc:270] Start()
0810 23:32:25.930602 (+ 441us) write_op.cc:276] Timestamp: P: 1786404745929988 usec, L: 0
0810 23:32:25.930687 (+ 85us) op_driver.cc:331] REPLICATION: starting
0810 23:32:25.932639 (+ 1952us) log.cc:842] Serialized 404 byte log entry
0810 23:32:26.160748 (+228109us) op_driver.cc:444] REPLICATION: finished
0810 23:32:26.171884 (+ 11136us) write_op.cc:301] APPLY: starting
0810 23:32:26.172003 (+ 119us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:26.172364 (+ 361us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:26.172646 (+ 282us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:26.173049 (+ 403us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:26.188332 (+ 15283us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:26.188513 (+ 181us) write_op.cc:312] APPLY: finished
0810 23:32:26.189177 (+ 664us) log.cc:842] Serialized 22 byte log entry
0810 23:32:26.198246 (+ 9069us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:26.198487 (+ 241us) write_op.cc:454] Released schema lock
0810 23:32:26.198731 (+ 244us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:26.200424 (+ 1693us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:26.200511 (+ 87us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":5318912,"child_traces":[["op",{"apply.queue_time_us":10725,"apply.run_cpu_time_us":4599,"apply.run_wall_time_us":29040,"num_ops":1,"prepare.queue_time_us":225814,"prepare.run_cpu_time_us":8410,"prepare.run_wall_time_us":9071,"raft.queue_time_us":256,"raft.run_cpu_time_us":35,"raft.run_wall_time_us":38,"replication_time_us":229787}]]}
W20260810 23:32:26.934015 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa1ea1ebbc1d4264a5c9f638092a5825 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:26.936720 13514 raft_consensus.cc:348] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:26.858003 13604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:26.939670 13514 raft_consensus.cc:374] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:26.991189 13514 raft_consensus.cc:729] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:26.976322 13526 ts_tablet_manager.cc:1918] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:26.995285 13526 log.cc:1196] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/b3ee96e60575425eb3e98ae1ab461180
I20260810 23:32:27.004340 13514 consensus_queue.cc:260] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:27.021123 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 56 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:27.037390 12228 master_service.cc:946] ReplaceTablet: received request to replace tablet df746466f2c9456d8261adb266a27aa8 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:27.054175 13526 ts_tablet_manager.cc:1939] T b3ee96e60575425eb3e98ae1ab461180 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:27.061086 13503 tablet_bootstrap.cc:654] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:27.068403 13604 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
W20260810 23:32:27.088694 13615 log.cc:924] Time spent T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.066s user 0.002s sys 0.000s
W20260810 23:32:27.102689 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 75ead79616d047f9972611623fc943b1 (table default.Table_e0603d4692934dce9e07597811ccad0a [id=ad699de357934075aee8d7152f8e130b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 75ead79616d047f9972611623fc943b1 consensus state has no leader
I20260810 23:32:27.115367 12235 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet e019ef94399b4e26babb2e94c1e4c6e9
W20260810 23:32:27.122336 12454 leader_election.cc:334] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:27.133185 13526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 13) took 1128 ms. Trace:
I20260810 23:32:27.137276 13509 tablet_bootstrap.cc:700] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b: Time spent opening tablet: real 0.105s user 0.005s sys 0.001s
I20260810 23:32:27.138221 13509 tablet_bootstrap.cc:654] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:27.123837 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 681674cb24274f10b200f264f155c2d1; no voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:27.146781 13604 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260810 23:32:27.155651 12698 leader_election.cc:341] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: df746466f2c9456d8261adb266a27aa8
I20260810 23:32:27.157824 13533 ts_tablet_manager.cc:1918] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:27.159192 13533 log.cc:1196] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/df746466f2c9456d8261adb266a27aa8
I20260810 23:32:27.157306 12698 leader_election.cc:302] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:27.182251 13533 ts_tablet_manager.cc:1939] T df746466f2c9456d8261adb266a27aa8 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
W20260810 23:32:27.188263 12454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 65978.5us
W20260810 23:32:27.200733 13213 consensus_peers.cc:487] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:27.217265 13555 raft_consensus.cc:504] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:27.134157 13526 rpcz_store.cc:276] 0810 23:32:26.005665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:26.006119 (+ 454us) service_pool.cc:227] Handling call
0810 23:32:26.347816 (+341697us) ts_tablet_manager.cc:1145] Deleting tablet b3ee96e60575425eb3e98ae1ab461180
0810 23:32:26.368146 (+ 20330us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:26.721221 (+353075us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:26.935673 (+214452us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.133027 (+197354us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":339232}
I20260810 23:32:27.229000 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet b3ee96e60575425eb3e98ae1ab461180 (table default.Table_e987219e3104474daa72d065991a72dd [id=d24331d84ad04768ab54058120b7e72f]) successfully deleted
I20260810 23:32:27.233332 13514 ts_tablet_manager.cc:1428] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.390s user 0.018s sys 0.015s
I20260810 23:32:27.248970 12234 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 65) took 1052 ms. Trace:
I20260810 23:32:27.264842 12229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 26b0c3f3068744e0899c7bb1e2823282
I20260810 23:32:27.281386 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151ba1dd24574739a9f0d33c2257235d" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:27.282835 13060 raft_consensus.cc:3037] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:27.290904 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b0c3f3068744e0899c7bb1e2823282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:26 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:27.107983 12223 catalog_manager.cc:5495] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } }
I20260810 23:32:27.308275 12235 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_b2d2da314c7e45c182359a84356a67f8" } new_table_name: "default.Table_3e7e62b6301644238020193af19fb270" modify_external_catalogs: true
I20260810 23:32:27.250298 12234 rpcz_store.cc:276] 0810 23:32:26.197034 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:26.200658 (+ 3624us) service_pool.cc:227] Handling call
0810 23:32:26.273522 (+ 72864us) spinlock_profiling.cc:243] Waited 56.2 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7156 00007f0ce6e24034 00007f0ce6e23fc2 00007f0ce6e23ef1 00007f0ce6e23e6c 00007f0ce6e23122 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:26.372976 (+ 99454us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:26.506575 (+133599us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:27.246223 (+739648us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:27.247753 (+ 1530us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:27.248754 (+ 1001us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:26.540727 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:26.542066 (+ 1339us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:26.542150 (+ 84us) write_op.cc:435] Acquired schema lock
0810 23:32:26.542200 (+ 50us) tablet.cc:660] Decoding operations
0810 23:32:26.543339 (+ 1139us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:26.543447 (+ 108us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:26.543519 (+ 72us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:26.543970 (+ 451us) tablet.cc:699] Row locks acquired
0810 23:32:26.543999 (+ 29us) write_op.cc:260] PREPARE: finished
0810 23:32:26.544684 (+ 685us) write_op.cc:270] Start()
0810 23:32:26.544913 (+ 229us) write_op.cc:276] Timestamp: P: 1786404746544545 usec, L: 0
0810 23:32:26.545112 (+ 199us) op_driver.cc:331] REPLICATION: starting
0810 23:32:26.546563 (+ 1451us) log.cc:842] Serialized 432 byte log entry
0810 23:32:27.233422 (+686859us) op_driver.cc:444] REPLICATION: finished
0810 23:32:27.239310 (+ 5888us) write_op.cc:301] APPLY: starting
0810 23:32:27.239446 (+ 136us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:27.239789 (+ 343us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:27.240259 (+ 470us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:27.241922 (+ 1663us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:27.242218 (+ 296us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:27.242298 (+ 80us) write_op.cc:312] APPLY: finished
0810 23:32:27.243101 (+ 803us) log.cc:842] Serialized 28 byte log entry
0810 23:32:27.245409 (+ 2308us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:27.245565 (+ 156us) write_op.cc:454] Released schema lock
0810 23:32:27.245664 (+ 99us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:27.247242 (+ 1578us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:27.247330 (+ 88us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":129185536,"child_traces":[["op",{"apply.queue_time_us":983,"apply.run_cpu_time_us":5838,"apply.run_wall_time_us":8635,"num_ops":2,"prepare.queue_time_us":24728,"prepare.run_cpu_time_us":5785,"prepare.run_wall_time_us":10370,"replication_time_us":688087}]]}
I20260810 23:32:27.326349 13555 leader_election.cc:288] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:27.344621 13533 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 12) took 2188 ms. Trace:
I20260810 23:32:27.355486 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b0c3f3068744e0899c7bb1e2823282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:26 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:27.355259 13533 rpcz_store.cc:276] 0810 23:32:25.156550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:25.156707 (+ 157us) service_pool.cc:227] Handling call
0810 23:32:26.327034 (+1170327us) ts_tablet_manager.cc:1145] Deleting tablet df746466f2c9456d8261adb266a27aa8
0810 23:32:26.327252 (+ 218us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:26.674646 (+347394us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.142260 (+467614us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.344502 (+202242us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1272,"tablet-delete.queue_time_us":1159262}
I20260810 23:32:27.368956 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet df746466f2c9456d8261adb266a27aa8 (table default.Table_4ba2f23aa6fb42398071dfe0239679f1 [id=48e13d1cc9d2459d96a57948f250d0c8]) successfully deleted
I20260810 23:32:27.375432 13514 tablet_bootstrap.cc:492] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:27.377168 13503 tablet_bootstrap.cc:492] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:27.378479 13503 ts_tablet_manager.cc:1397] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.720s user 0.021s sys 0.031s
I20260810 23:32:27.379340 13533 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 13) took 2116 ms. Trace:
I20260810 23:32:27.384292 13533 rpcz_store.cc:276] 0810 23:32:25.263536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:25.263779 (+ 243us) service_pool.cc:227] Handling call
0810 23:32:27.367326 (+2103547us) ts_tablet_manager.cc:1145] Deleting tablet 7be44f795bcd4cdcbdb2f3a03ce5253f
0810 23:32:27.367471 (+ 145us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:27.379171 (+ 11700us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2101568}
W20260810 23:32:27.388239 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7be44f795bcd4cdcbdb2f3a03ce5253f already in progress: creating tablet
I20260810 23:32:27.392473 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b0c3f3068744e0899c7bb1e2823282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:26 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:27.393180 13533 tablet_replica.cc:330] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:27.394129 12772 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5e80d295c94a24854747e468d54de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:27.395686 13533 raft_consensus.cc:2229] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:27.358667 13623 raft_consensus.cc:2726] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:27.400074 13532 ts_tablet_manager.cc:1918] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:27.380525 13509 tablet_bootstrap.cc:492] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:27.402033 13509 ts_tablet_manager.cc:1397] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.752s user 0.020s sys 0.021s
I20260810 23:32:27.400190 13503 raft_consensus.cc:348] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:27.403131 13503 raft_consensus.cc:374] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:27.404953 13503 raft_consensus.cc:729] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:27.380105 13060 raft_consensus.cc:2454] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:27.401937 13532 log.cc:1196] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/af05dca78b474e8189fecae9f32a711a
I20260810 23:32:27.412900 13212 leader_election.cc:302] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:27.412945 13532 ts_tablet_manager.cc:1939] T af05dca78b474e8189fecae9f32a711a P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:27.415493 13555 raft_consensus.cc:2781] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:27.416465 13555 raft_consensus.cc:686] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:27.418757 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_705b0928d5734aa5afa8f557d194f1e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:27.420630 13532 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 13) took 1612 ms. Trace:
W20260810 23:32:27.421921 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_705b0928d5734aa5afa8f557d194f1e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:27.413455 13503 consensus_queue.cc:260] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:27.438496 13503 ts_tablet_manager.cc:1428] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.058s user 0.011s sys 0.004s
I20260810 23:32:27.446004 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151ba1dd24574739a9f0d33c2257235d" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:27.463402 13503 tablet_bootstrap.cc:492] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:27.421515 13532 rpcz_store.cc:276] 0810 23:32:25.808971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:25.839549 (+ 30578us) service_pool.cc:227] Handling call
0810 23:32:26.548493 (+708944us) ts_tablet_manager.cc:1145] Deleting tablet af05dca78b474e8189fecae9f32a711a
0810 23:32:26.548755 (+ 262us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:27.099565 (+550810us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.399606 (+300041us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.420345 (+ 20739us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":706412}
I20260810 23:32:27.419044 13555 consensus_queue.cc:237] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:27.255632 13526 tablet_replica.cc:330] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
W20260810 23:32:27.484551 13213 leader_election.cc:341] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:27.487202 12768 consensus_meta.cc:287] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.527s user 0.022s sys 0.000s
I20260810 23:32:27.489916 13533 pending_rounds.cc:62] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b: Trying to abort 1 pending ops.
I20260810 23:32:27.491914 13533 pending_rounds.cc:68] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317113843247026176 op_type: NO_OP noop_request { }
I20260810 23:32:27.493263 13533 raft_consensus.cc:2866] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:32:27.487023 13509 raft_consensus.cc:348] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:27.494644 13509 raft_consensus.cc:374] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:27.495217 13509 raft_consensus.cc:729] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:27.496804 13509 consensus_queue.cc:260] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:27.498715 13533 raft_consensus.cc:2258] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:27.504863 13509 ts_tablet_manager.cc:1428] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.102s user 0.008s sys 0.006s
I20260810 23:32:27.542440 13533 ts_tablet_manager.cc:1905] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:27.385293 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 230088us
I20260810 23:32:27.576766 13532 tablet_replica.cc:330] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:27.334227 12223 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 75ead79616d047f9972611623fc943b1 (table default.Table_e0603d4692934dce9e07597811ccad0a [id=ad699de357934075aee8d7152f8e130b]). Expected version 1 got 0
I20260810 23:32:27.330592 12229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_edb5813389c6488ea72bae6a8373f405" } modify_external_catalogs: true
I20260810 23:32:27.525525 12237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 151ba1dd24574739a9f0d33c2257235d
W20260810 23:32:27.541307 12226 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5b08ed744b3420f98d241ad96bca6d0 has no consensus state
I20260810 23:32:27.601135 12226 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 71) took 1115 ms. Trace:
I20260810 23:32:27.599417 13526 raft_consensus.cc:2229] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:27.605515 13526 raft_consensus.cc:2258] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:27.550303 12234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e987219e3104474daa72d065991a72dd" } modify_external_catalogs: true
I20260810 23:32:27.608918 12234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e987219e3104474daa72d065991a72dd" } modify_external_catalogs: true
I20260810 23:32:27.558527 12236 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 70) took 1090 ms. Trace:
I20260810 23:32:27.611047 12768 tablet_service.cc:1514] Processing DeleteTablet for tablet 151ba1dd24574739a9f0d33c2257235d with delete_type TABLET_DATA_DELETED (replaced by tablet fd110664bac34f52bfe922a3a76c6290) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:27.606686 12229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_edb5813389c6488ea72bae6a8373f405" } modify_external_catalogs: true
I20260810 23:32:27.630200 13533 ts_tablet_manager.cc:1918] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:27.631546 13526 ts_tablet_manager.cc:1905] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:27.631770 13533 log.cc:1196] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/af05dca78b474e8189fecae9f32a711a
I20260810 23:32:27.622365 12237 catalog_manager.cc:6565] ReplaceTablet: tablet 151ba1dd24574739a9f0d33c2257235d deleted and replaced by tablet fd110664bac34f52bfe922a3a76c6290
I20260810 23:32:27.643015 12237 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 64) took 1577 ms. Trace:
I20260810 23:32:27.643560 13533 ts_tablet_manager.cc:1939] T af05dca78b474e8189fecae9f32a711a P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:27.649998 13503 tablet_bootstrap.cc:654] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:27.650242 13514 tablet_bootstrap.cc:654] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:27.663837 13533 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 14) took 1867 ms. Trace:
W20260810 23:32:27.648837 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 164305us
I20260810 23:32:27.665093 13533 rpcz_store.cc:276] 0810 23:32:25.796702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:25.797029 (+ 327us) service_pool.cc:227] Handling call
0810 23:32:27.392729 (+1595700us) ts_tablet_manager.cc:1145] Deleting tablet af05dca78b474e8189fecae9f32a711a
0810 23:32:27.392866 (+ 137us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:27.568789 (+175923us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.629139 (+ 60350us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.663694 (+ 34555us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1594491}
I20260810 23:32:27.667608 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 151ba1dd24574739a9f0d33c2257235d with delete_type TABLET_DATA_DELETED (replaced by tablet fd110664bac34f52bfe922a3a76c6290) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:27.672380 13533 tablet_replica.cc:330] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:27.674844 13533 raft_consensus.cc:2229] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:27.677481 13533 raft_consensus.cc:2258] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:27.677116 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet af05dca78b474e8189fecae9f32a711a (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]) successfully deleted
I20260810 23:32:27.682938 13533 ts_tablet_manager.cc:1905] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:27.692268 13532 raft_consensus.cc:2229] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:27.693403 13532 raft_consensus.cc:2258] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:27.602002 12226 rpcz_store.cc:276] 0810 23:32:26.486043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:26.780182 (+294139us) service_pool.cc:227] Handling call
0810 23:32:26.846497 (+ 66315us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:26.850483 (+ 3986us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:26.922238 (+ 71755us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:26.923023 (+ 785us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:27.539845 (+616822us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:27.600980 (+ 61135us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:26.941036 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:26.942185 (+ 1149us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:26.942231 (+ 46us) write_op.cc:435] Acquired schema lock
0810 23:32:26.942263 (+ 32us) tablet.cc:660] Decoding operations
0810 23:32:26.942884 (+ 621us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:26.942954 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:26.942994 (+ 40us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:26.943343 (+ 349us) tablet.cc:699] Row locks acquired
0810 23:32:26.943360 (+ 17us) write_op.cc:260] PREPARE: finished
0810 23:32:26.943602 (+ 242us) write_op.cc:270] Start()
0810 23:32:26.943750 (+ 148us) write_op.cc:276] Timestamp: P: 1786404746943033 usec, L: 0
0810 23:32:26.943768 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 23:32:26.944954 (+ 1186us) log.cc:842] Serialized 404 byte log entry
0810 23:32:27.512931 (+567977us) op_driver.cc:444] REPLICATION: finished
0810 23:32:27.513804 (+ 873us) write_op.cc:301] APPLY: starting
0810 23:32:27.514009 (+ 205us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:27.514307 (+ 298us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:27.514706 (+ 399us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:27.515224 (+ 518us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:27.515746 (+ 522us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:27.515849 (+ 103us) write_op.cc:312] APPLY: finished
0810 23:32:27.536533 (+ 20684us) log.cc:842] Serialized 22 byte log entry
0810 23:32:27.538856 (+ 2323us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:27.539016 (+ 160us) write_op.cc:454] Released schema lock
0810 23:32:27.539080 (+ 64us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:27.542409 (+ 3329us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:27.542524 (+ 115us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":280,"apply.run_cpu_time_us":4857,"apply.run_wall_time_us":29393,"num_ops":1,"prepare.queue_time_us":13878,"prepare.run_cpu_time_us":4105,"prepare.run_wall_time_us":7408,"replication_time_us":568803}]]}
I20260810 23:32:27.704533 13532 ts_tablet_manager.cc:1905] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:27.643707 12237 rpcz_store.cc:276] 0810 23:32:26.065612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:26.065940 (+ 328us) service_pool.cc:227] Handling call
0810 23:32:27.642861 (+1576921us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:26.821262 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:26.821758 (+ 496us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:26.821785 (+ 27us) write_op.cc:435] Acquired schema lock
0810 23:32:26.821799 (+ 14us) tablet.cc:660] Decoding operations
0810 23:32:26.822244 (+ 445us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:26.822428 (+ 184us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:26.822453 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:26.822834 (+ 381us) tablet.cc:699] Row locks acquired
0810 23:32:26.822849 (+ 15us) write_op.cc:260] PREPARE: finished
0810 23:32:26.823367 (+ 518us) write_op.cc:270] Start()
0810 23:32:26.823528 (+ 161us) write_op.cc:276] Timestamp: P: 1786404746822791 usec, L: 0
0810 23:32:26.823561 (+ 33us) op_driver.cc:331] REPLICATION: starting
0810 23:32:26.825756 (+ 2195us) log.cc:842] Serialized 630 byte log entry
0810 23:32:27.507784 (+682028us) op_driver.cc:444] REPLICATION: finished
0810 23:32:27.508694 (+ 910us) write_op.cc:301] APPLY: starting
0810 23:32:27.508812 (+ 118us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:27.509379 (+ 567us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:27.509811 (+ 432us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:27.510969 (+ 1158us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:27.511571 (+ 602us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:27.511747 (+ 176us) write_op.cc:312] APPLY: finished
0810 23:32:27.519600 (+ 7853us) log.cc:842] Serialized 28 byte log entry
0810 23:32:27.522103 (+ 2503us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:27.522424 (+ 321us) write_op.cc:454] Released schema lock
0810 23:32:27.522541 (+ 117us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:27.532149 (+ 9608us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:27.532208 (+ 59us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":245,"apply.run_cpu_time_us":6499,"apply.run_wall_time_us":24055,"mutex_wait_us":50028,"num_ops":2,"prepare.queue_time_us":56573,"prepare.run_cpu_time_us":4148,"prepare.run_wall_time_us":8123,"replication_time_us":664291}]]}
I20260810 23:32:27.611789 12236 rpcz_store.cc:276] 0810 23:32:26.483804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:26.678062 (+194258us) service_pool.cc:227] Handling call
0810 23:32:26.834574 (+156512us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:26.917902 (+ 83328us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:27.556170 (+638268us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:27.557290 (+ 1120us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:27.558325 (+ 1035us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:27.000358 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:27.000991 (+ 633us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:27.001033 (+ 42us) write_op.cc:435] Acquired schema lock
0810 23:32:27.001055 (+ 22us) tablet.cc:660] Decoding operations
0810 23:32:27.001608 (+ 553us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:27.001674 (+ 66us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:27.001709 (+ 35us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:27.002094 (+ 385us) tablet.cc:699] Row locks acquired
0810 23:32:27.002119 (+ 25us) write_op.cc:260] PREPARE: finished
0810 23:32:27.002253 (+ 134us) write_op.cc:270] Start()
0810 23:32:27.002347 (+ 94us) write_op.cc:276] Timestamp: P: 1786404747001711 usec, L: 0
0810 23:32:27.002363 (+ 16us) op_driver.cc:331] REPLICATION: starting
0810 23:32:27.003139 (+ 776us) log.cc:842] Serialized 432 byte log entry
0810 23:32:27.517415 (+514276us) op_driver.cc:444] REPLICATION: finished
0810 23:32:27.533405 (+ 15990us) write_op.cc:301] APPLY: starting
0810 23:32:27.533544 (+ 139us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:27.533881 (+ 337us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:27.534127 (+ 246us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:27.543913 (+ 9786us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:27.544446 (+ 533us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:27.544602 (+ 156us) write_op.cc:312] APPLY: finished
0810 23:32:27.552460 (+ 7858us) log.cc:842] Serialized 28 byte log entry
0810 23:32:27.555138 (+ 2678us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:27.555411 (+ 273us) write_op.cc:454] Released schema lock
0810 23:32:27.555495 (+ 84us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:27.560064 (+ 4569us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:27.560164 (+ 100us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":15472,"apply.run_cpu_time_us":6459,"apply.run_wall_time_us":27220,"num_ops":2,"prepare.queue_time_us":269,"prepare.run_cpu_time_us":3285,"prepare.run_wall_time_us":7496,"replication_time_us":514822,"thread_start_us":28834,"threads_started":1}]]}
I20260810 23:32:27.735682 13533 ts_tablet_manager.cc:1918] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:27.737087 13533 log.cc:1196] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/b3ee96e60575425eb3e98ae1ab461180
I20260810 23:32:27.736974 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_fa01ae9b566c41d1accafece4db355bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:27.739095 13533 ts_tablet_manager.cc:1939] T b3ee96e60575425eb3e98ae1ab461180 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:27.743770 13503 tablet_bootstrap.cc:492] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:27.745121 13503 ts_tablet_manager.cc:1397] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.283s user 0.025s sys 0.003s
I20260810 23:32:27.769049 12224 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 72) took 1269 ms. Trace:
I20260810 23:32:27.770462 12237 master_service.cc:946] ReplaceTablet: received request to replace tablet ee5e80d295c94a24854747e468d54de6 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:27.769887 12224 rpcz_store.cc:276] 0810 23:32:26.499428 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:26.796636 (+297208us) service_pool.cc:227] Handling call
0810 23:32:26.935153 (+138517us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:27.030595 (+ 95442us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:27.766911 (+736316us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:27.767711 (+ 800us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:27.768891 (+ 1180us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:27.085997 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:27.087005 (+ 1008us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:27.087069 (+ 64us) write_op.cc:435] Acquired schema lock
0810 23:32:27.087103 (+ 34us) tablet.cc:660] Decoding operations
0810 23:32:27.087851 (+ 748us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:27.087937 (+ 86us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:27.087982 (+ 45us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:27.088448 (+ 466us) tablet.cc:699] Row locks acquired
0810 23:32:27.088474 (+ 26us) write_op.cc:260] PREPARE: finished
0810 23:32:27.088693 (+ 219us) write_op.cc:270] Start()
0810 23:32:27.088848 (+ 155us) write_op.cc:276] Timestamp: P: 1786404747088121 usec, L: 0
0810 23:32:27.088875 (+ 27us) op_driver.cc:331] REPLICATION: starting
0810 23:32:27.089939 (+ 1064us) log.cc:842] Serialized 432 byte log entry
0810 23:32:27.547189 (+457250us) op_driver.cc:444] REPLICATION: finished
0810 23:32:27.560993 (+ 13804us) write_op.cc:301] APPLY: starting
0810 23:32:27.561234 (+ 241us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:27.561767 (+ 533us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:27.562125 (+ 358us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:27.764096 (+201971us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:27.764469 (+ 373us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:27.764587 (+ 118us) write_op.cc:312] APPLY: finished
0810 23:32:27.765094 (+ 507us) log.cc:842] Serialized 28 byte log entry
0810 23:32:27.766477 (+ 1383us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:27.766579 (+ 102us) write_op.cc:454] Released schema lock
0810 23:32:27.766618 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:27.766982 (+ 364us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:27.767030 (+ 48us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":13451,"apply.run_cpu_time_us":5876,"apply.run_wall_time_us":206430,"num_ops":2,"prepare.queue_time_us":11048,"prepare.run_cpu_time_us":4097,"prepare.run_wall_time_us":6911,"replication_time_us":458169}]]}
I20260810 23:32:27.780491 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet af05dca78b474e8189fecae9f32a711a (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]) successfully deleted
I20260810 23:32:27.781752 13555 raft_consensus.cc:482] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:27.782583 13555 raft_consensus.cc:504] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:27.790903 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be44f795bcd4cdcbdb2f3a03ce5253f" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:27.793068 13514 tablet_bootstrap.cc:492] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:27.793828 13514 ts_tablet_manager.cc:1397] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.419s user 0.019s sys 0.033s
W20260810 23:32:27.794291 13213 leader_election.cc:341] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:27.793861 13503 raft_consensus.cc:348] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:27.794979 13503 raft_consensus.cc:374] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:27.795262 13503 raft_consensus.cc:729] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:27.795974 13555 leader_election.cc:288] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
W20260810 23:32:27.797410 12235 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b9b01e7b3093472bbeb08e930a6ef1b6 has no consensus state
I20260810 23:32:27.804572 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet 151ba1dd24574739a9f0d33c2257235d with delete_type TABLET_DATA_DELETED (replaced by tablet fd110664bac34f52bfe922a3a76c6290) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:27.807440 13025 consensus_meta.cc:287] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.650s user 0.011s sys 0.011s
I20260810 23:32:27.805863 13503 consensus_queue.cc:260] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:27.813402 13614 consensus_queue.cc:1035] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:27.814153 13503 ts_tablet_manager.cc:1428] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.068s user 0.008s sys 0.004s
I20260810 23:32:27.818344 13025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 16) took 1080 ms. Trace:
I20260810 23:32:27.819196 13514 raft_consensus.cc:348] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:27.828229 13514 raft_consensus.cc:374] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:27.828689 13514 raft_consensus.cc:729] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:27.832013 13514 consensus_queue.cc:260] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:27.833803 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa01ae9b566c41d1accafece4db355bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:27.835526 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be44f795bcd4cdcbdb2f3a03ce5253f" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:27.837196 13062 raft_consensus.cc:2454] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:27.827644 13526 ts_tablet_manager.cc:1918] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:27.839983 13212 leader_election.cc:302] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:27.840508 13526 log.cc:1196] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/df746466f2c9456d8261adb266a27aa8
I20260810 23:32:27.842056 13555 raft_consensus.cc:2781] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:27.842621 13555 raft_consensus.cc:482] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:27.843261 13555 raft_consensus.cc:3037] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:27.855859 13526 ts_tablet_manager.cc:1939] T df746466f2c9456d8261adb266a27aa8 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:27.859697 13526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 14) took 1573 ms. Trace:
I20260810 23:32:27.861099 13514 ts_tablet_manager.cc:1428] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.067s user 0.013s sys 0.000s
I20260810 23:32:27.864342 13509 tablet_bootstrap.cc:492] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:27.819193 13025 rpcz_store.cc:276] 0810 23:32:26.737959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:26.738385 (+ 426us) service_pool.cc:227] Handling call
0810 23:32:26.743536 (+ 5151us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:26.743813 (+ 277us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:27.156536 (+412723us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.816872 (+660336us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:27.818172 (+ 1300us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-open.queue_time_us":45958}
I20260810 23:32:27.866269 13532 ts_tablet_manager.cc:1918] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:27.867092 13532 log.cc:1196] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/b3ee96e60575425eb3e98ae1ab461180
I20260810 23:32:27.869282 13514 tablet_bootstrap.cc:492] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:27.874082 13526 rpcz_store.cc:276] 0810 23:32:26.286725 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:26.287087 (+ 362us) service_pool.cc:227] Handling call
0810 23:32:27.255145 (+968058us) ts_tablet_manager.cc:1145] Deleting tablet df746466f2c9456d8261adb266a27aa8
0810 23:32:27.255300 (+ 155us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:27.722264 (+466964us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.827170 (+104906us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.859616 (+ 32446us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":116148,"tablet-delete.queue_time_us":962499}
I20260810 23:32:27.876667 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet df746466f2c9456d8261adb266a27aa8 (table default.Table_4ba2f23aa6fb42398071dfe0239679f1 [id=48e13d1cc9d2459d96a57948f250d0c8]) successfully deleted
I20260810 23:32:27.883294 13509 tablet_bootstrap.cc:654] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:27.746084 13533 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 15) took 1732 ms. Trace:
I20260810 23:32:27.884583 13533 rpcz_store.cc:276] 0810 23:32:26.013721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:26.014002 (+ 281us) service_pool.cc:227] Handling call
0810 23:32:27.671781 (+1657779us) ts_tablet_manager.cc:1145] Deleting tablet b3ee96e60575425eb3e98ae1ab461180
0810 23:32:27.671986 (+ 205us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:27.709543 (+ 37557us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.734949 (+ 25406us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.745931 (+ 10982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1655697}
W20260810 23:32:27.886745 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 26b0c3f3068744e0899c7bb1e2823282 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26b0c3f3068744e0899c7bb1e2823282 already in progress: creating tablet
I20260810 23:32:27.889480 13533 tablet_replica.cc:330] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:27.892087 13533 raft_consensus.cc:2229] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:27.893903 13526 tablet_replica.cc:330] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:27.894244 13533 raft_consensus.cc:2258] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:27.898000 13532 ts_tablet_manager.cc:1939] T b3ee96e60575425eb3e98ae1ab461180 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:27.898322 13526 raft_consensus.cc:2229] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:27.901382 13526 raft_consensus.cc:2258] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:27.905586 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet b3ee96e60575425eb3e98ae1ab461180 (table default.Table_e987219e3104474daa72d065991a72dd [id=d24331d84ad04768ab54058120b7e72f]) successfully deleted
I20260810 23:32:27.912439 12237 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_69a8d6b34dd84459a083c452460bd345" } new_table_name: "default.Table_089a1834ee404319b9d766443f10704f" modify_external_catalogs: true
I20260810 23:32:27.914404 13533 ts_tablet_manager.cc:1905] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:27.910010 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 26b0c3f3068744e0899c7bb1e2823282 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26b0c3f3068744e0899c7bb1e2823282 already in progress: creating tablet
I20260810 23:32:27.914467 13514 tablet_bootstrap.cc:654] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:27.918090 13532 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 14) took 1912 ms. Trace:
I20260810 23:32:27.919205 13532 rpcz_store.cc:276] 0810 23:32:26.006253 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:26.010168 (+ 3915us) service_pool.cc:227] Handling call
0810 23:32:27.576479 (+1566311us) ts_tablet_manager.cc:1145] Deleting tablet b3ee96e60575425eb3e98ae1ab461180
0810 23:32:27.576548 (+ 69us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:27.807883 (+231335us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.852938 (+ 45055us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:27.917896 (+ 64958us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":113302,"tablet-delete.queue_time_us":1565054}
I20260810 23:32:27.923353 13532 tablet_replica.cc:330] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:27.923715 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet b3ee96e60575425eb3e98ae1ab461180 (table default.Table_e987219e3104474daa72d065991a72dd [id=d24331d84ad04768ab54058120b7e72f]) successfully deleted
I20260810 23:32:27.919868 13526 ts_tablet_manager.cc:1905] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:27.941747 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet af05dca78b474e8189fecae9f32a711a with delete_type TABLET_DATA_DELETED (replaced by tablet 217a7582902244f9b0ca65be1c20af22) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:27.941157 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_9e4b92615b33447baec585c92237ff05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:27.950222 12768 tablet_service.cc:1514] Processing DeleteTablet for tablet af05dca78b474e8189fecae9f32a711a with delete_type TABLET_DATA_DELETED (replaced by tablet 217a7582902244f9b0ca65be1c20af22) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:27.957880 13532 raft_consensus.cc:2229] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:27.959404 13532 raft_consensus.cc:2258] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:27.965025 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e4b92615b33447baec585c92237ff05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:27.977751 12223 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 9) took 1030 ms (client timeout 1000 ms). Trace:
I20260810 23:32:27.981348 13532 ts_tablet_manager.cc:1905] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:27.982442 12226 master_service.cc:946] ReplaceTablet: received request to replace tablet 75ead79616d047f9972611623fc943b1 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:27.983810 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_e6944762a73d4c80ae0866fad8945b41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:27.978559 12223 rpcz_store.cc:269] 0810 23:32:26.948139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:26.949533 (+ 1394us) service_pool.cc:227] Handling call
0810 23:32:27.977607 (+1028074us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:27.693020 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:27.693614 (+ 594us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:27.693642 (+ 28us) write_op.cc:435] Acquired schema lock
0810 23:32:27.693657 (+ 15us) tablet.cc:660] Decoding operations
0810 23:32:27.694125 (+ 468us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:27.694194 (+ 69us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:27.694221 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:27.694529 (+ 308us) tablet.cc:699] Row locks acquired
0810 23:32:27.694551 (+ 22us) write_op.cc:260] PREPARE: finished
0810 23:32:27.694905 (+ 354us) write_op.cc:270] Start()
0810 23:32:27.695082 (+ 177us) write_op.cc:276] Timestamp: P: 1786404747694767 usec, L: 0
0810 23:32:27.695116 (+ 34us) op_driver.cc:331] REPLICATION: starting
0810 23:32:27.696886 (+ 1770us) log.cc:842] Serialized 535 byte log entry
0810 23:32:27.873942 (+177056us) op_driver.cc:444] REPLICATION: finished
0810 23:32:27.874639 (+ 697us) write_op.cc:301] APPLY: starting
0810 23:32:27.874764 (+ 125us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:27.874981 (+ 217us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:27.875203 (+ 222us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:27.875473 (+ 270us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:27.875681 (+ 208us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:27.875725 (+ 44us) write_op.cc:312] APPLY: finished
0810 23:32:27.875953 (+ 228us) log.cc:842] Serialized 22 byte log entry
0810 23:32:27.890808 (+ 14855us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:27.890963 (+ 155us) write_op.cc:454] Released schema lock
0810 23:32:27.891025 (+ 62us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:27.891521 (+ 496us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:27.891564 (+ 43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":13721,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":331,"apply.run_cpu_time_us":2820,"apply.run_wall_time_us":17242,"mutex_wait_us":92241,"num_ops":1,"prepare.queue_time_us":106083,"prepare.run_cpu_time_us":15849,"prepare.run_wall_time_us":93174,"raft.queue_time_us":20587,"raft.run_cpu_time_us":21,"raft.run_wall_time_us":23,"replication_time_us":178630,"thread_start_us":83535,"threads_started":1}]]}
I20260810 23:32:28.009744 13514 tablet_bootstrap.cc:492] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
W20260810 23:32:27.987331 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6944762a73d4c80ae0866fad8945b41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:27.995836 13555 raft_consensus.cc:504] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:28.012130 13514 ts_tablet_manager.cc:1397] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.143s user 0.013s sys 0.013s
I20260810 23:32:28.012570 12223 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_aa1ea1ebbc1d4264a5c9f638092a5825" } new_table_name: "default.Table_275ea19d22624a12a6b3b2c8e985c4af" modify_external_catalogs: true
I20260810 23:32:28.052155 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be44f795bcd4cdcbdb2f3a03ce5253f" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
W20260810 23:32:28.056447 13213 leader_election.cc:341] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:28.058959 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be44f795bcd4cdcbdb2f3a03ce5253f" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:28.060139 13062 raft_consensus.cc:3037] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:28.087225 13514 raft_consensus.cc:348] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:28.105031 13514 raft_consensus.cc:374] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:28.112108 13514 raft_consensus.cc:729] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:28.072481 12234 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 922ce728af9740108dc959b3155ffca7
I20260810 23:32:28.090117 12240 master_service.cc:946] ReplaceTablet: received request to replace tablet 75ead79616d047f9972611623fc943b1 from {username='slave'} at 127.0.0.1:34742
W20260810 23:32:28.111985 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 74103.7us
I20260810 23:32:28.128111 13514 consensus_queue.cc:260] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:28.135152 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet b9b01e7b3093472bbeb08e930a6ef1b6 (table default.Table_3e7e62b6301644238020193af19fb270 [id=b5cc65e503f7413fbade0c94e2e74b14], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b9b01e7b3093472bbeb08e930a6ef1b6 has no consensus state
I20260810 23:32:28.135179 13555 leader_election.cc:288] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
W20260810 23:32:28.143183 12237 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f791b2df0524997ab1c53a9e5cb39ca has no consensus state
I20260810 23:32:28.154855 13625 raft_consensus.cc:482] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:28.155778 13625 raft_consensus.cc:504] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:28.161105 13625 leader_election.cc:288] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:28.161738 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ead79616d047f9972611623fc943b1" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:28.164855 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ead79616d047f9972611623fc943b1" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:28.170315 12700 leader_election.cc:302] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:28.174841 13625 raft_consensus.cc:2726] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:28.177175 13514 ts_tablet_manager.cc:1428] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.160s user 0.014s sys 0.004s
I20260810 23:32:28.157979 13509 tablet_bootstrap.cc:492] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:28.210765 13532 ts_tablet_manager.cc:1918] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:28.212679 13532 log.cc:1196] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/26b0c3f3068744e0899c7bb1e2823282
I20260810 23:32:28.213142 13509 ts_tablet_manager.cc:1397] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.349s user 0.025s sys 0.006s
I20260810 23:32:28.214249 13532 ts_tablet_manager.cc:1939] T 26b0c3f3068744e0899c7bb1e2823282 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:28.220528 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 26b0c3f3068744e0899c7bb1e2823282 (table default.Table_bf3c79e609d24e7cb76242c93decb907 [id=afcd22d5c266477cbc0f2160f852346b]) successfully deleted
I20260810 23:32:28.224107 13532 tablet_replica.cc:330] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:28.225994 13532 raft_consensus.cc:2229] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:28.227118 13514 tablet_bootstrap.cc:492] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:28.232168 13532 pending_rounds.cc:62] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3: Trying to abort 1 pending ops.
I20260810 23:32:28.232981 13532 pending_rounds.cc:68] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317113846072950784 op_type: NO_OP noop_request { }
I20260810 23:32:28.203366 13661 tablet.cc:1722] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:28.229736 13662 tablet.cc:1722] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:28.236002 13532 raft_consensus.cc:2866] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:32:28.236308 13062 raft_consensus.cc:2454] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:28.236618 13532 raft_consensus.cc:2258] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:28.241636 13526 ts_tablet_manager.cc:1918] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:28.247828 13526 log.cc:1196] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/ee5e80d295c94a24854747e468d54de6
I20260810 23:32:28.249782 13212 leader_election.cc:302] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:28.251089 13532 ts_tablet_manager.cc:1905] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:28.254272 13526 ts_tablet_manager.cc:1939] T ee5e80d295c94a24854747e468d54de6 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:28.252451 13652 raft_consensus.cc:2781] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:28.255255 13652 raft_consensus.cc:686] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:28.256747 13652 consensus_queue.cc:237] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:28.218901 13509 raft_consensus.cc:348] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:28.267426 13509 raft_consensus.cc:374] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:28.269629 13509 raft_consensus.cc:729] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:28.278618 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_7c223ce60be04a498d08155325785819"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:28.279062 13509 consensus_queue.cc:260] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:28.283458 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c223ce60be04a498d08155325785819 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:28.307058 13509 ts_tablet_manager.cc:1428] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.093s user 0.012s sys 0.000s
I20260810 23:32:28.327544 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet ee5e80d295c94a24854747e468d54de6 (table default.Table_8b5a278156ad4137bf0ea955a9bb88cb [id=c4b2684e4dd74e0f9170cffcd54c2cb9]) successfully deleted
W20260810 23:32:28.356578 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 151ba1dd24574739a9f0d33c2257235d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 151ba1dd24574739a9f0d33c2257235d already in progress: creating tablet
I20260810 23:32:28.369801 13526 tablet_replica.cc:330] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:28.381260 13532 ts_tablet_manager.cc:1918] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:28.382181 13532 log.cc:1196] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/151ba1dd24574739a9f0d33c2257235d
I20260810 23:32:28.382128 13526 raft_consensus.cc:2229] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:28.382573 13533 ts_tablet_manager.cc:1918] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:28.383698 13526 raft_consensus.cc:2258] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:28.383975 13532 ts_tablet_manager.cc:1939] T 151ba1dd24574739a9f0d33c2257235d P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:28.310442 13514 tablet_bootstrap.cc:654] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:28.394532 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 151ba1dd24574739a9f0d33c2257235d (table default.Table_0af7ca97e51c47a29d5dd8867b152066 [id=32144cd75df84cfabce64657802c111a]) successfully deleted
I20260810 23:32:28.387830 13526 ts_tablet_manager.cc:1905] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:28.398417 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet af05dca78b474e8189fecae9f32a711a because the tablet was not found. No further retry: Not found: Tablet not found: af05dca78b474e8189fecae9f32a711a
I20260810 23:32:28.383595 13533 log.cc:1196] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/151ba1dd24574739a9f0d33c2257235d
I20260810 23:32:28.405660 13533 ts_tablet_manager.cc:1939] T 151ba1dd24574739a9f0d33c2257235d P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:28.426162 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 151ba1dd24574739a9f0d33c2257235d (table default.Table_0af7ca97e51c47a29d5dd8867b152066 [id=32144cd75df84cfabce64657802c111a]) successfully deleted
I20260810 23:32:28.450716 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_43485ff6f5de42c7be9cf286a18133fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:28.455003 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43485ff6f5de42c7be9cf286a18133fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:28.465713 12223 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79da9583d3524347a9be09e35affaaad has no consensus state
I20260810 23:32:28.479631 13291 tablet_service.cc:1467] Processing CreateTablet for tablet b9b01e7b3093472bbeb08e930a6ef1b6 (DEFAULT_TABLE table=default.Table_3e7e62b6301644238020193af19fb270 [id=b5cc65e503f7413fbade0c94e2e74b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.481874 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b01e7b3093472bbeb08e930a6ef1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.477977 12223 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_85ae9e6bdba74ff0bcb1027b39707b5e" } modify_external_catalogs: true
I20260810 23:32:28.482534 12223 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_85ae9e6bdba74ff0bcb1027b39707b5e" } modify_external_catalogs: true
W20260810 23:32:28.489835 12228 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 15) took 1406 ms (client timeout 1000 ms). Trace:
I20260810 23:32:28.490880 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet af05dca78b474e8189fecae9f32a711a with delete_type TABLET_DATA_DELETED (replaced by tablet 217a7582902244f9b0ca65be1c20af22) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:28.490741 12228 rpcz_store.cc:269] 0810 23:32:27.083964 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:27.087093 (+ 3129us) service_pool.cc:227] Handling call
0810 23:32:27.324455 (+237362us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:28.489650 (+1165195us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":397961984}
I20260810 23:32:28.501564 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_d318aced7c0540b3b58280f0285ecf87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:28.504988 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet 151ba1dd24574739a9f0d33c2257235d with delete_type TABLET_DATA_DELETED (replaced by tablet fd110664bac34f52bfe922a3a76c6290) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:28.509128 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet af05dca78b474e8189fecae9f32a711a because the tablet was not found. No further retry: Not found: Tablet not found: af05dca78b474e8189fecae9f32a711a
I20260810 23:32:28.510527 12768 tablet_service.cc:1467] Processing CreateTablet for tablet b9b01e7b3093472bbeb08e930a6ef1b6 (DEFAULT_TABLE table=default.Table_3e7e62b6301644238020193af19fb270 [id=b5cc65e503f7413fbade0c94e2e74b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.503705 13032 tablet_service.cc:1467] Processing CreateTablet for tablet b9b01e7b3093472bbeb08e930a6ef1b6 (DEFAULT_TABLE table=default.Table_3e7e62b6301644238020193af19fb270 [id=b5cc65e503f7413fbade0c94e2e74b14]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:28.514919 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d318aced7c0540b3b58280f0285ecf87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:28.515388 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b01e7b3093472bbeb08e930a6ef1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.516670 12772 tablet_service.cc:1467] Processing CreateTablet for tablet 5df6b412b60f400b829b6042460aa195 (DEFAULT_TABLE table=default.Table_10156b698d054ae8845a67255a9daf3d [id=c48d5d1f759b42f3a5c5a6c64588b30f]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:28.521225 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 151ba1dd24574739a9f0d33c2257235d because the tablet was not found. No further retry: Not found: Tablet not found: 151ba1dd24574739a9f0d33c2257235d
I20260810 23:32:28.525789 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b01e7b3093472bbeb08e930a6ef1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.543747 12772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df6b412b60f400b829b6042460aa195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.549979 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 5df6b412b60f400b829b6042460aa195 (DEFAULT_TABLE table=default.Table_10156b698d054ae8845a67255a9daf3d [id=c48d5d1f759b42f3a5c5a6c64588b30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.551066 13025 tablet_service.cc:1467] Processing CreateTablet for tablet 5df6b412b60f400b829b6042460aa195 (DEFAULT_TABLE table=default.Table_10156b698d054ae8845a67255a9daf3d [id=c48d5d1f759b42f3a5c5a6c64588b30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.553557 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df6b412b60f400b829b6042460aa195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.553858 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df6b412b60f400b829b6042460aa195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.580128 13514 tablet_bootstrap.cc:492] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:28.581532 13514 ts_tablet_manager.cc:1397] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.402s user 0.017s sys 0.010s
I20260810 23:32:28.620910 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 88c53785fdd24df79f138df6176e75d2 (DEFAULT_TABLE table=default.Table_90dfb8b49e7d41fc8267c506f1e3f57e [id=71c9424f327040a68889fc257935c535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.622581 13514 raft_consensus.cc:348] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:28.625897 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c53785fdd24df79f138df6176e75d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.638651 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 88c53785fdd24df79f138df6176e75d2 (DEFAULT_TABLE table=default.Table_90dfb8b49e7d41fc8267c506f1e3f57e [id=71c9424f327040a68889fc257935c535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.641316 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c53785fdd24df79f138df6176e75d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.645527 13665 tablet_bootstrap.cc:492] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:28.683285 13031 tablet_service.cc:1467] Processing CreateTablet for tablet 88c53785fdd24df79f138df6176e75d2 (DEFAULT_TABLE table=default.Table_90dfb8b49e7d41fc8267c506f1e3f57e [id=71c9424f327040a68889fc257935c535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.686857 12769 tablet_service.cc:1467] Processing CreateTablet for tablet e5b08ed744b3420f98d241ad96bca6d0 (DEFAULT_TABLE table=default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.689193 13286 tablet_service.cc:1467] Processing CreateTablet for tablet e5b08ed744b3420f98d241ad96bca6d0 (DEFAULT_TABLE table=default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.693041 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b08ed744b3420f98d241ad96bca6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.697028 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b08ed744b3420f98d241ad96bca6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.696111 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c53785fdd24df79f138df6176e75d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.757573 13026 tablet_service.cc:1467] Processing CreateTablet for tablet e5b08ed744b3420f98d241ad96bca6d0 (DEFAULT_TABLE table=default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.761621 13026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b08ed744b3420f98d241ad96bca6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.762758 12767 tablet_service.cc:1467] Processing CreateTablet for tablet 585baf44aca142199fbc268ff3bcd8f4 (DEFAULT_TABLE table=default.Table_edb5813389c6488ea72bae6a8373f405 [id=5c3bdcf03c3d43cbbbd2b53dcdc8fa95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.766134 12767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585baf44aca142199fbc268ff3bcd8f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.761513 12235 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_3e7e62b6301644238020193af19fb270" } new_table_name: "default.Table_21d34a701c494bc580c412d025ddf4dd" modify_external_catalogs: true
I20260810 23:32:28.724543 13514 raft_consensus.cc:374] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:28.801108 13514 raft_consensus.cc:729] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:28.799631 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 585baf44aca142199fbc268ff3bcd8f4 (DEFAULT_TABLE table=default.Table_edb5813389c6488ea72bae6a8373f405 [id=5c3bdcf03c3d43cbbbd2b53dcdc8fa95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.805563 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585baf44aca142199fbc268ff3bcd8f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.843192 13283 tablet_service.cc:1467] Processing CreateTablet for tablet dce0ec83ed8943a0b0788505bb75d1df (DEFAULT_TABLE table=default.Table_fd3d342652da4d5baad3608431df6041 [id=e462d54230344223a457f708501d2d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.802526 13514 consensus_queue.cc:260] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:28.872227 13022 tablet_service.cc:1467] Processing CreateTablet for tablet dce0ec83ed8943a0b0788505bb75d1df (DEFAULT_TABLE table=default.Table_fd3d342652da4d5baad3608431df6041 [id=e462d54230344223a457f708501d2d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.874943 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce0ec83ed8943a0b0788505bb75d1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.878733 13023 tablet_service.cc:1467] Processing CreateTablet for tablet 585baf44aca142199fbc268ff3bcd8f4 (DEFAULT_TABLE table=default.Table_edb5813389c6488ea72bae6a8373f405 [id=5c3bdcf03c3d43cbbbd2b53dcdc8fa95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.893002 12806 raft_consensus.cc:3037] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:28.896749 13021 tablet_service.cc:1467] Processing CreateTablet for tablet 217a7582902244f9b0ca65be1c20af22 (DEFAULT_TABLE table=default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.907410 12766 tablet_service.cc:1467] Processing CreateTablet for tablet dce0ec83ed8943a0b0788505bb75d1df (DEFAULT_TABLE table=default.Table_fd3d342652da4d5baad3608431df6041 [id=e462d54230344223a457f708501d2d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.922279 13021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217a7582902244f9b0ca65be1c20af22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.913421 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 217a7582902244f9b0ca65be1c20af22 (DEFAULT_TABLE table=default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:28.932286 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce0ec83ed8943a0b0788505bb75d1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.934849 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585baf44aca142199fbc268ff3bcd8f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.940824 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce0ec83ed8943a0b0788505bb75d1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.949059 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217a7582902244f9b0ca65be1c20af22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.951920 13514 ts_tablet_manager.cc:1428] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.370s user 0.013s sys 0.007s
I20260810 23:32:28.969396 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 217a7582902244f9b0ca65be1c20af22 (DEFAULT_TABLE table=default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:28.972189 12215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260810 23:32:28.979132 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217a7582902244f9b0ca65be1c20af22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:29.001865 13665 tablet_bootstrap.cc:654] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:29.072682 13671 raft_consensus.cc:482] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:29.098858 13514 tablet_bootstrap.cc:492] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:29.074637 13671 raft_consensus.cc:504] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:29.139673 12226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 75ead79616d047f9972611623fc943b1
W20260810 23:32:29.155364 12218 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 79da9583d3524347a9be09e35affaaad (table default.Table_275ea19d22624a12a6b3b2c8e985c4af [id=5b4476e15a7e410599b70f84ab267de9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 79da9583d3524347a9be09e35affaaad has no consensus state
I20260810 23:32:29.164686 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ead79616d047f9972611623fc943b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 840400119d20424ea0f03f2bff550d44) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:29.167930 12240 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 85) took 1092 ms. Trace:
I20260810 23:32:29.168649 12240 rpcz_store.cc:276] 0810 23:32:28.089452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.089778 (+ 326us) service_pool.cc:227] Handling call
0810 23:32:28.168418 (+ 78640us) spinlock_profiling.cc:243] Waited 40.5 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e9b81 00007f0ce6e44d37 00007f0ce6e44cc2 00007f0ce6e44bf1 00007f0ce6e44b6c 00007f0ce6e43e22 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:29.167756 (+999338us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93129600}
W20260810 23:32:29.179687 13213 consensus_peers.cc:487] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:29.185644 12237 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 29) took 1022 ms (client timeout 1000 ms). Trace:
I20260810 23:32:29.200345 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ead79616d047f9972611623fc943b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 840400119d20424ea0f03f2bff550d44) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:29.208391 12226 catalog_manager.cc:6565] ReplaceTablet: tablet 75ead79616d047f9972611623fc943b1 deleted and replaced by tablet 840400119d20424ea0f03f2bff550d44
I20260810 23:32:29.216070 12226 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 82) took 1263 ms. Trace:
I20260810 23:32:29.216991 13679 tablet_replica.cc:330] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:29.229863 13679 raft_consensus.cc:2229] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:29.238095 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ead79616d047f9972611623fc943b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 840400119d20424ea0f03f2bff550d44) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:29.241097 12698 leader_election.cc:341] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 151ba1dd24574739a9f0d33c2257235d
I20260810 23:32:29.253799 13671 leader_election.cc:288] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:29.265785 13679 raft_consensus.cc:2258] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:29.212235 12237 rpcz_store.cc:269] 0810 23:32:28.163506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.190414 (+ 26908us) service_pool.cc:227] Handling call
0810 23:32:29.185464 (+995050us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260810 23:32:29.291008 13680 tablet_replica.cc:330] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:29.292236 13679 ts_tablet_manager.cc:1905] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:29.294689 13680 raft_consensus.cc:2229] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:29.302726 13526 ts_tablet_manager.cc:1918] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:29.320050 13652 consensus_queue.cc:1035] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:32:29.326706 12700 leader_election.cc:341] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Not found: Tablet not found: 151ba1dd24574739a9f0d33c2257235d
I20260810 23:32:29.328110 12700 leader_election.cc:302] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:29.331498 13671 raft_consensus.cc:2726] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:29.297737 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 151ba1dd24574739a9f0d33c2257235d with delete_type TABLET_DATA_DELETED (replaced by tablet fd110664bac34f52bfe922a3a76c6290) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:29.335837 12229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 585baf44aca142199fbc268ff3bcd8f4
I20260810 23:32:29.313380 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:29.345957 13665 tablet_bootstrap.cc:492] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:29.345566 12229 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 75) took 2022 ms. Trace:
I20260810 23:32:29.347216 13665 ts_tablet_manager.cc:1397] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.702s user 0.020s sys 0.043s
I20260810 23:32:29.348621 13526 log.cc:1196] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/af05dca78b474e8189fecae9f32a711a
I20260810 23:32:29.351164 13526 ts_tablet_manager.cc:1939] T af05dca78b474e8189fecae9f32a711a P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:29.359808 12772 tablet_service.cc:1514] Processing DeleteTablet for tablet 585baf44aca142199fbc268ff3bcd8f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:27 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:29.378026 12228 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 89) took 1019 ms. Trace:
I20260810 23:32:29.378909 13674 raft_consensus.cc:482] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:32:29.414846 12236 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 16) took 1171 ms (client timeout 1000 ms). Trace:
I20260810 23:32:29.465795 13514 tablet_bootstrap.cc:700] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3: Time spent opening tablet: real 0.148s user 0.002s sys 0.001s
I20260810 23:32:29.422520 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 585baf44aca142199fbc268ff3bcd8f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:27 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:29.317394 13032 consensus_meta.cc:287] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.502s user 0.009s sys 0.012s
I20260810 23:32:29.365234 13680 raft_consensus.cc:2258] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:29.478323 12770 consensus_meta.cc:287] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.540s user 0.010s sys 0.011s
I20260810 23:32:29.515748 13665 raft_consensus.cc:348] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:29.551775 13665 raft_consensus.cc:374] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:29.553051 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 585baf44aca142199fbc268ff3bcd8f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:27 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:29.554618 13665 raft_consensus.cc:729] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
W20260810 23:32:29.492813 12236 rpcz_store.cc:269] 0810 23:32:28.242427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.242743 (+ 316us) service_pool.cc:227] Handling call
0810 23:32:29.413883 (+1171140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44587,"reported_tablets":3}
I20260810 23:32:29.582010 13665 consensus_queue.cc:260] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:29.623772 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_969f8ae489e24f139b305369d629d7af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:29.621095 13514 tablet_bootstrap.cc:654] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:29.692087 13665 ts_tablet_manager.cc:1428] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.332s user 0.049s sys 0.007s
I20260810 23:32:29.641247 12223 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 3b71ea20eaf84cf2bff6ef11b44caa58
I20260810 23:32:29.705940 12223 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 88) took 1364 ms. Trace:
W20260810 23:32:29.691238 12218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.188:45477, user_credentials={real_user=slave}} blocked reactor thread for 140434us
I20260810 23:32:29.265062 12226 rpcz_store.cc:276] 0810 23:32:27.952713 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:27.982171 (+ 29458us) service_pool.cc:227] Handling call
0810 23:32:29.215858 (+1233687us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:28.484501 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:28.485119 (+ 618us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:28.485182 (+ 63us) write_op.cc:435] Acquired schema lock
0810 23:32:28.485208 (+ 26us) tablet.cc:660] Decoding operations
0810 23:32:28.485774 (+ 566us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:28.485848 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:28.485890 (+ 42us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:28.486283 (+ 393us) tablet.cc:699] Row locks acquired
0810 23:32:28.486320 (+ 37us) write_op.cc:260] PREPARE: finished
0810 23:32:28.486661 (+ 341us) write_op.cc:270] Start()
0810 23:32:28.486824 (+ 163us) write_op.cc:276] Timestamp: P: 1786404748486559 usec, L: 0
0810 23:32:28.486873 (+ 49us) op_driver.cc:331] REPLICATION: starting
0810 23:32:28.487803 (+ 930us) log.cc:842] Serialized 664 byte log entry
0810 23:32:29.112265 (+624462us) op_driver.cc:444] REPLICATION: finished
0810 23:32:29.114498 (+ 2233us) write_op.cc:301] APPLY: starting
0810 23:32:29.114663 (+ 165us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:29.115000 (+ 337us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:29.115228 (+ 228us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:29.117415 (+ 2187us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:29.118074 (+ 659us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:29.118213 (+ 139us) write_op.cc:312] APPLY: finished
0810 23:32:29.118724 (+ 511us) log.cc:842] Serialized 28 byte log entry
0810 23:32:29.132626 (+ 13902us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:29.132756 (+ 130us) write_op.cc:454] Released schema lock
0810 23:32:29.132890 (+ 134us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:29.160910 (+ 28020us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:29.161001 (+ 91us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1769,"apply.run_cpu_time_us":6670,"apply.run_wall_time_us":47252,"num_ops":2,"prepare.queue_time_us":201,"prepare.run_cpu_time_us":3964,"prepare.run_wall_time_us":12399,"raft.queue_time_us":19369,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":43,"replication_time_us":625167}]]}
W20260810 23:32:29.710719 12235 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b9b01e7b3093472bbeb08e930a6ef1b6 consensus state has no leader
I20260810 23:32:29.721573 12226 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_705b0928d5734aa5afa8f557d194f1e2" } new_table_name: "default.Table_41341e2688304faa88c0ccc969e5faa7" modify_external_catalogs: true
I20260810 23:32:29.692648 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 151ba1dd24574739a9f0d33c2257235d with delete_type TABLET_DATA_DELETED (replaced by tablet fd110664bac34f52bfe922a3a76c6290) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:29.727265 12235 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 90) took 1367 ms. Trace:
I20260810 23:32:29.800943 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:29.805866 13665 tablet_bootstrap.cc:492] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
W20260810 23:32:29.810031 12234 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 10) took 1681 ms (client timeout 1000 ms). Trace:
W20260810 23:32:29.814780 12234 rpcz_store.cc:269] 0810 23:32:28.128678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.128950 (+ 272us) service_pool.cc:227] Handling call
0810 23:32:28.164236 (+ 35286us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:29.809639 (+1645403us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":4,"spinlock_wait_cycles":72756096}
I20260810 23:32:29.507378 13674 raft_consensus.cc:504] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:29.810833 12770 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 22) took 1181 ms. Trace:
I20260810 23:32:29.830080 13526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 19) took 1881 ms. Trace:
W20260810 23:32:29.826392 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_969f8ae489e24f139b305369d629d7af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:29.708250 12223 rpcz_store.cc:276] 0810 23:32:28.341533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.467847 (+126314us) service_pool.cc:227] Handling call
0810 23:32:28.483581 (+ 15734us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:28.532802 (+ 49221us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:28.533148 (+ 346us) catalog_manager.cc:2761] Locking tablets
0810 23:32:28.534310 (+ 1162us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:29.631910 (+1097600us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:29.633847 (+ 1937us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:29.635475 (+ 1628us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:29.705402 (+ 69927us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:28.651795 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:28.652559 (+ 764us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:28.652613 (+ 54us) write_op.cc:435] Acquired schema lock
0810 23:32:28.652644 (+ 31us) tablet.cc:660] Decoding operations
0810 23:32:28.653433 (+ 789us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:28.653507 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:28.653542 (+ 35us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:28.654354 (+ 812us) tablet.cc:699] Row locks acquired
0810 23:32:28.654388 (+ 34us) write_op.cc:260] PREPARE: finished
0810 23:32:28.654624 (+ 236us) write_op.cc:270] Start()
0810 23:32:28.654858 (+ 234us) write_op.cc:276] Timestamp: P: 1786404748654517 usec, L: 0
0810 23:32:28.654886 (+ 28us) op_driver.cc:331] REPLICATION: starting
0810 23:32:28.687855 (+ 32969us) log.cc:842] Serialized 532 byte log entry
0810 23:32:29.365281 (+677426us) op_driver.cc:444] REPLICATION: finished
0810 23:32:29.398479 (+ 33198us) write_op.cc:301] APPLY: starting
0810 23:32:29.399305 (+ 826us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:29.529328 (+130023us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:29.561389 (+ 32061us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:29.562919 (+ 1530us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:29.563377 (+ 458us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:29.563646 (+ 269us) write_op.cc:312] APPLY: finished
0810 23:32:29.609736 (+ 46090us) log.cc:842] Serialized 28 byte log entry
0810 23:32:29.610133 (+ 397us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:29.630802 (+ 20669us) write_op.cc:454] Released schema lock
0810 23:32:29.631226 (+ 424us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:29.637532 (+ 6306us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:29.637644 (+ 112us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4322,"child_traces":[["op",{"apply.queue_time_us":31946,"apply.run_cpu_time_us":13854,"apply.run_wall_time_us":240049,"num_ops":2,"prepare.queue_time_us":680,"prepare.run_cpu_time_us":4420,"prepare.run_wall_time_us":50315,"replication_time_us":710182,"thread_start_us":56762,"threads_started":1}]]}
I20260810 23:32:29.846112 12770 rpcz_store.cc:276] 0810 23:32:28.616608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.618048 (+ 1440us) service_pool.cc:227] Handling call
0810 23:32:28.622856 (+ 4808us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:28.623069 (+ 213us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:28.936796 (+313727us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:29.736595 (+799799us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:29.797683 (+ 61088us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:29.848992 13526 rpcz_store.cc:276] 0810 23:32:27.946769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:27.949959 (+ 3190us) service_pool.cc:227] Handling call
0810 23:32:28.369177 (+419218us) ts_tablet_manager.cc:1145] Deleting tablet af05dca78b474e8189fecae9f32a711a
0810 23:32:28.369393 (+ 216us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:28.922811 (+553418us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:29.302157 (+379346us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:29.828136 (+525979us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2249,"tablet-delete.queue_time_us":418243}
I20260810 23:32:29.941658 13526 tablet_replica.cc:330] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:29.950851 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b0c3f3068744e0899c7bb1e2823282" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
W20260810 23:32:29.965866 12958 leader_election.cc:341] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:29.971850 12952 leader_election.cc:341] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 26b0c3f3068744e0899c7bb1e2823282
I20260810 23:32:29.975332 12952 leader_election.cc:302] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:29.954594 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_b6ab31c2c3cb48fe84b341db2abba9ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:29.995223 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b0c3f3068744e0899c7bb1e2823282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:26 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:30.009327 13695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e964330cf113403a8aa5b73ddd3b31d0)
I20260810 23:32:30.012001 13681 raft_consensus.cc:2726] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:30.013471 13680 ts_tablet_manager.cc:1905] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:30.017910 13674 leader_election.cc:288] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
W20260810 23:32:30.026166 13471 log.cc:924] Time spent T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Append to log took a long time: real 0.216s user 0.003s sys 0.000s
I20260810 23:32:29.379956 12228 rpcz_store.cc:276] 0810 23:32:28.358518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.500661 (+142143us) service_pool.cc:227] Handling call
0810 23:32:28.515901 (+ 15240us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:28.519861 (+ 3960us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:29.375186 (+855325us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:29.376590 (+ 1404us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:29.377847 (+ 1257us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:28.556323 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:28.556882 (+ 559us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:28.556912 (+ 30us) write_op.cc:435] Acquired schema lock
0810 23:32:28.556935 (+ 23us) tablet.cc:660] Decoding operations
0810 23:32:28.557470 (+ 535us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:28.557533 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:28.557579 (+ 46us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:28.557993 (+ 414us) tablet.cc:699] Row locks acquired
0810 23:32:28.558030 (+ 37us) write_op.cc:260] PREPARE: finished
0810 23:32:28.558280 (+ 250us) write_op.cc:270] Start()
0810 23:32:28.558460 (+ 180us) write_op.cc:276] Timestamp: P: 1786404748558177 usec, L: 0
0810 23:32:28.558491 (+ 31us) op_driver.cc:331] REPLICATION: starting
0810 23:32:28.559669 (+ 1178us) log.cc:842] Serialized 432 byte log entry
0810 23:32:29.364245 (+804576us) op_driver.cc:444] REPLICATION: finished
0810 23:32:29.367122 (+ 2877us) write_op.cc:301] APPLY: starting
0810 23:32:29.367287 (+ 165us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:29.367715 (+ 428us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:29.368394 (+ 679us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:29.370085 (+ 1691us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:29.370777 (+ 692us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:29.370877 (+ 100us) write_op.cc:312] APPLY: finished
0810 23:32:29.371318 (+ 441us) log.cc:842] Serialized 28 byte log entry
0810 23:32:29.374128 (+ 2810us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:29.374493 (+ 365us) write_op.cc:454] Released schema lock
0810 23:32:29.374585 (+ 92us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2168,"apply.run_cpu_time_us":8906,"apply.run_wall_time_us":64834,"num_ops":2,"prepare.queue_time_us":14853,"prepare.run_cpu_time_us":3817,"prepare.run_wall_time_us":95150,"replication_time_us":805572}]]}
W20260810 23:32:30.014191 12767 consensus_meta.cc:287] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.693s user 0.031s sys 0.005s
W20260810 23:32:29.991747 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6ab31c2c3cb48fe84b341db2abba9ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:30.041096 12767 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 24) took 1283 ms. Trace:
I20260810 23:32:29.984102 13032 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 21) took 1432 ms. Trace:
W20260810 23:32:30.020256 13688 log.cc:924] Time spent T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260810 23:32:30.058758 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet af05dca78b474e8189fecae9f32a711a (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]) successfully deleted
I20260810 23:32:30.059098 12224 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 87) took 1720 ms. Trace:
I20260810 23:32:29.350924 12229 rpcz_store.cc:276] 0810 23:32:27.322903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:27.328669 (+ 5766us) service_pool.cc:227] Handling call
0810 23:32:27.626645 (+297976us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:27.636184 (+ 9539us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:27.636489 (+ 305us) catalog_manager.cc:2761] Locking tablets
0810 23:32:28.484857 (+848368us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:29.330186 (+845329us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:29.335298 (+ 5112us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:29.335616 (+ 318us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:29.345396 (+ 9780us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:28.523651 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:28.524368 (+ 717us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:28.524410 (+ 42us) write_op.cc:435] Acquired schema lock
0810 23:32:28.524445 (+ 35us) tablet.cc:660] Decoding operations
0810 23:32:28.525123 (+ 678us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:28.525198 (+ 75us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:28.525237 (+ 39us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:28.525784 (+ 547us) tablet.cc:699] Row locks acquired
0810 23:32:28.525812 (+ 28us) write_op.cc:260] PREPARE: finished
0810 23:32:28.526050 (+ 238us) write_op.cc:270] Start()
0810 23:32:28.526223 (+ 173us) write_op.cc:276] Timestamp: P: 1786404748525963 usec, L: 0
0810 23:32:28.526254 (+ 31us) op_driver.cc:331] REPLICATION: starting
0810 23:32:28.527342 (+ 1088us) log.cc:842] Serialized 733 byte log entry
0810 23:32:29.113031 (+585689us) op_driver.cc:444] REPLICATION: finished
0810 23:32:29.162296 (+ 49265us) write_op.cc:301] APPLY: starting
0810 23:32:29.162450 (+ 154us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:29.162791 (+ 341us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:29.163031 (+ 240us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:29.232063 (+ 69032us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:29.232476 (+ 413us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:29.232731 (+ 255us) write_op.cc:312] APPLY: finished
0810 23:32:29.233457 (+ 726us) log.cc:842] Serialized 28 byte log entry
0810 23:32:29.300992 (+ 67535us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:29.301201 (+ 209us) write_op.cc:454] Released schema lock
0810 23:32:29.303678 (+ 2477us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:29.333238 (+ 29560us) write_op.cc:489] Releasing partition, row and schema locks
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48745,"apply.run_cpu_time_us":6079,"apply.run_wall_time_us":238692,"num_ops":2,"prepare.queue_time_us":22887,"prepare.run_cpu_time_us":4278,"prepare.run_wall_time_us":8086,"raft.queue_time_us":3691,"raft.run_cpu_time_us":44,"raft.run_wall_time_us":47,"replication_time_us":586537,"thread_start_us":250195,"threads_started":1}]]}
I20260810 23:32:30.076359 13695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:30.059856 12224 rpcz_store.cc:276] 0810 23:32:28.338635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.449505 (+110870us) service_pool.cc:227] Handling call
0810 23:32:28.456002 (+ 6497us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:28.461920 (+ 5918us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:30.049577 (+1587657us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:30.058440 (+ 8863us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:30.058997 (+ 557us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:28.706391 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:28.711029 (+ 4638us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:28.717447 (+ 6418us) write_op.cc:435] Acquired schema lock
0810 23:32:28.717504 (+ 57us) tablet.cc:660] Decoding operations
0810 23:32:28.768505 (+ 51001us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:28.768600 (+ 95us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:28.768625 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:28.768979 (+ 354us) tablet.cc:699] Row locks acquired
0810 23:32:28.769003 (+ 24us) write_op.cc:260] PREPARE: finished
0810 23:32:28.769460 (+ 457us) write_op.cc:270] Start()
0810 23:32:28.769683 (+ 223us) write_op.cc:276] Timestamp: P: 1786404748769180 usec, L: 0
0810 23:32:28.769720 (+ 37us) op_driver.cc:331] REPLICATION: starting
0810 23:32:28.771126 (+ 1406us) log.cc:842] Serialized 432 byte log entry
0810 23:32:29.689963 (+918837us) op_driver.cc:444] REPLICATION: finished
0810 23:32:29.693177 (+ 3214us) write_op.cc:301] APPLY: starting
0810 23:32:29.693747 (+ 570us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:29.695063 (+ 1316us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:29.729222 (+ 34159us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:29.735125 (+ 5903us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:29.764558 (+ 29433us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:29.765249 (+ 691us) write_op.cc:312] APPLY: finished
0810 23:32:29.766511 (+ 1262us) log.cc:842] Serialized 28 byte log entry
0810 23:32:29.767713 (+ 1202us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:30.047911 (+280198us) write_op.cc:454] Released schema lock
0810 23:32:30.048109 (+ 198us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:30.051108 (+ 2999us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:30.051168 (+ 60us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2054,"apply.run_cpu_time_us":15887,"apply.run_wall_time_us":359899,"num_ops":2,"prepare.queue_time_us":13862,"prepare.run_cpu_time_us":4446,"prepare.run_wall_time_us":65015,"replication_time_us":920090}]]}
W20260810 23:32:30.084923 12240 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 11) took 1017 ms (client timeout 1000 ms). Trace:
W20260810 23:32:30.085615 12240 rpcz_store.cc:269] 0810 23:32:29.067956 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:29.173094 (+105138us) service_pool.cc:227] Handling call
0810 23:32:29.615990 (+442896us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:30.084758 (+468768us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":861814656}
I20260810 23:32:30.042052 12767 rpcz_store.cc:276] 0810 23:32:28.758752 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.759514 (+ 762us) service_pool.cc:227] Handling call
0810 23:32:28.764954 (+ 5440us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:28.765198 (+ 244us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:29.320108 (+554910us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.034598 (+714490us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:30.040810 (+ 6212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:30.044333 13032 rpcz_store.cc:276] 0810 23:32:28.500809 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.501074 (+ 265us) service_pool.cc:227] Handling call
0810 23:32:28.513993 (+ 12919us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:28.514157 (+ 164us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:28.813471 (+299314us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:29.515221 (+701750us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:29.932771 (+417550us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33590656,"tablet-open.queue_time_us":286972,"thread_start_us":283816,"threads_started":1}
I20260810 23:32:30.125564 13665 tablet_bootstrap.cc:654] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:30.096951 12768 consensus_meta.cc:287] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.756s user 0.019s sys 0.020s
I20260810 23:32:30.132467 12223 master_service.cc:946] ReplaceTablet: received request to replace tablet b9b01e7b3093472bbeb08e930a6ef1b6 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:30.153862 13514 tablet_bootstrap.cc:492] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:30.155186 13514 ts_tablet_manager.cc:1397] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 1.057s user 0.027s sys 0.024s
I20260810 23:32:30.159840 13679 ts_tablet_manager.cc:1918] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:30.161068 13679 log.cc:1196] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/75ead79616d047f9972611623fc943b1
I20260810 23:32:30.163558 13679 ts_tablet_manager.cc:1939] T 75ead79616d047f9972611623fc943b1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:30.164131 13691 tablet_bootstrap.cc:492] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
W20260810 23:32:30.181059 13025 consensus_meta.cc:287] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 1.108s user 0.030s sys 0.007s
I20260810 23:32:30.181077 12768 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 26) took 1270 ms. Trace:
W20260810 23:32:30.095139 12766 consensus_meta.cc:287] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.532s user 0.032s sys 0.003s
I20260810 23:32:30.186659 12766 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 25) took 1352 ms. Trace:
I20260810 23:32:30.187870 12766 rpcz_store.cc:276] 0810 23:32:28.834228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.897545 (+ 63317us) service_pool.cc:227] Handling call
0810 23:32:28.929300 (+ 31755us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:28.929614 (+ 314us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:29.338222 (+408608us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.185451 (+847229us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:30.186509 (+ 1058us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:30.182169 12768 rpcz_store.cc:276] 0810 23:32:28.911367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.911666 (+ 299us) service_pool.cc:227] Handling call
0810 23:32:28.929917 (+ 18251us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:28.930132 (+ 215us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:29.299133 (+369001us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.134095 (+834962us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:30.180873 (+ 46778us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:30.216013 13679 tablet_replica.cc:330] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:30.228571 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 75ead79616d047f9972611623fc943b1 (table default.Table_e0603d4692934dce9e07597811ccad0a [id=ad699de357934075aee8d7152f8e130b]) successfully deleted
I20260810 23:32:30.219202 13679 raft_consensus.cc:2229] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:30.234136 13514 raft_consensus.cc:348] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:30.238140 13679 raft_consensus.cc:2258] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:30.251659 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_e8c26e9362704b6fb84bba331ef685ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:30.231642 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 585baf44aca142199fbc268ff3bcd8f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 585baf44aca142199fbc268ff3bcd8f4 already in progress: creating tablet
I20260810 23:32:30.235651 13514 raft_consensus.cc:374] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:30.260759 13514 raft_consensus.cc:729] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:30.264217 12228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e6944762a73d4c80ae0866fad8945b41" } modify_external_catalogs: true
I20260810 23:32:30.266460 12228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e6944762a73d4c80ae0866fad8945b41" } modify_external_catalogs: true
I20260810 23:32:30.274663 13025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 22) took 1726 ms. Trace:
I20260810 23:32:30.275851 13025 rpcz_store.cc:276] 0810 23:32:28.548484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.548968 (+ 484us) service_pool.cc:227] Handling call
0810 23:32:28.552877 (+ 3909us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:28.553098 (+ 221us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:29.062257 (+509159us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.252496 (+1190239us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:30.274425 (+ 21929us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:30.281694 13679 ts_tablet_manager.cc:1905] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:30.290164 13665 tablet_bootstrap.cc:492] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:30.262534 13514 consensus_queue.cc:260] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:30.286239 12214 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 52178.7us
W20260810 23:32:30.290174 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8c26e9362704b6fb84bba331ef685ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:30.291756 13665 ts_tablet_manager.cc:1397] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.544s user 0.030s sys 0.024s
W20260810 23:32:30.302356 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 151ba1dd24574739a9f0d33c2257235d because the tablet was not found. No further retry: Not found: Tablet not found: 151ba1dd24574739a9f0d33c2257235d
I20260810 23:32:30.316668 13514 ts_tablet_manager.cc:1428] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.161s user 0.014s sys 0.008s
W20260810 23:32:30.264226 13031 consensus_meta.cc:287] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.580s user 0.018s sys 0.011s
W20260810 23:32:30.255220 13026 consensus_meta.cc:287] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.704s user 0.024s sys 0.004s
I20260810 23:32:30.343922 13665 raft_consensus.cc:348] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:30.345919 13665 raft_consensus.cc:374] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:30.346563 13665 raft_consensus.cc:729] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:30.349428 13031 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 23) took 1669 ms. Trace:
I20260810 23:32:30.354977 13026 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 24) took 1602 ms. Trace:
I20260810 23:32:30.351082 13031 rpcz_store.cc:276] 0810 23:32:28.680327 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.680969 (+ 642us) service_pool.cc:227] Handling call
0810 23:32:28.685834 (+ 4865us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:28.694076 (+ 8242us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:29.456669 (+762593us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.322024 (+865355us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:30.349144 (+ 27120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:30.365090 13665 consensus_queue.cc:260] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:30.356995 13026 rpcz_store.cc:276] 0810 23:32:28.752367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.753001 (+ 634us) service_pool.cc:227] Handling call
0810 23:32:28.759875 (+ 6874us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:28.760285 (+ 410us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:29.462286 (+702001us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.337218 (+874932us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:30.354608 (+ 17390us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:30.392452 13695 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:30.396366 13021 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 27) took 1503 ms. Trace:
I20260810 23:32:30.401582 13514 tablet_bootstrap.cc:492] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:32:30.410524 12217 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 103491us
I20260810 23:32:30.416925 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 81 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:30.427879 13526 raft_consensus.cc:2229] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:30.428567 12806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.11.226.131:56203 (request call id 10) took 1591 ms. Trace:
I20260810 23:32:30.430872 13526 raft_consensus.cc:2258] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:30.317056 12224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_9e4b92615b33447baec585c92237ff05" } modify_external_catalogs: true
I20260810 23:32:30.397648 13021 rpcz_store.cc:276] 0810 23:32:28.893334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.893996 (+ 662us) service_pool.cc:227] Handling call
0810 23:32:28.902291 (+ 8295us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:28.902631 (+ 340us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:29.964767 (+1062136us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.379416 (+414649us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:30.396171 (+ 16755us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:30.436698 12224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_9e4b92615b33447baec585c92237ff05" } modify_external_catalogs: true
W20260810 23:32:30.440497 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet b9b01e7b3093472bbeb08e930a6ef1b6 (table default.Table_21d34a701c494bc580c412d025ddf4dd [id=b5cc65e503f7413fbade0c94e2e74b14], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet b9b01e7b3093472bbeb08e930a6ef1b6 consensus state has no leader
I20260810 23:32:30.467727 13679 ts_tablet_manager.cc:1918] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:30.467080 13022 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 26) took 1631 ms. Trace:
I20260810 23:32:30.469820 13679 log.cc:1196] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/7be44f795bcd4cdcbdb2f3a03ce5253f
I20260810 23:32:30.472759 13679 ts_tablet_manager.cc:1939] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:30.475283 13023 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 25) took 1664 ms. Trace:
I20260810 23:32:29.798655 12235 rpcz_store.cc:276] 0810 23:32:28.358983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.759873 (+400890us) service_pool.cc:227] Handling call
0810 23:32:28.767037 (+ 7164us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:28.774562 (+ 7525us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:28.776274 (+ 1712us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:28.780288 (+ 4014us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:29.709290 (+929002us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:29.726585 (+ 17295us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:28.796528 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:28.797373 (+ 845us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:28.797429 (+ 56us) write_op.cc:435] Acquired schema lock
0810 23:32:28.797443 (+ 14us) tablet.cc:660] Decoding operations
0810 23:32:28.798031 (+ 588us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:28.798108 (+ 77us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:28.798164 (+ 56us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:28.798652 (+ 488us) tablet.cc:699] Row locks acquired
0810 23:32:28.798734 (+ 82us) write_op.cc:260] PREPARE: finished
0810 23:32:28.799113 (+ 379us) write_op.cc:270] Start()
0810 23:32:28.799300 (+ 187us) write_op.cc:276] Timestamp: P: 1786404748798909 usec, L: 0
0810 23:32:28.799338 (+ 38us) op_driver.cc:331] REPLICATION: starting
0810 23:32:28.800726 (+ 1388us) log.cc:842] Serialized 404 byte log entry
0810 23:32:29.690304 (+889578us) op_driver.cc:444] REPLICATION: finished
0810 23:32:29.697592 (+ 7288us) write_op.cc:301] APPLY: starting
0810 23:32:29.697900 (+ 308us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:29.698838 (+ 938us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:29.706148 (+ 7310us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:29.706762 (+ 614us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:29.707420 (+ 658us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:29.707746 (+ 326us) write_op.cc:312] APPLY: finished
0810 23:32:29.708305 (+ 559us) log.cc:842] Serialized 22 byte log entry
0810 23:32:29.708488 (+ 183us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:29.708561 (+ 73us) write_op.cc:454] Released schema lock
0810 23:32:29.708595 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:29.750111 (+ 41516us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:29.750671 (+ 560us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6448,"apply.run_cpu_time_us":8868,"apply.run_wall_time_us":85978,"num_ops":1,"prepare.queue_time_us":13027,"prepare.run_cpu_time_us":4934,"prepare.run_wall_time_us":4957,"replication_time_us":890844}]]}
I20260810 23:32:30.469650 13022 rpcz_store.cc:276] 0810 23:32:28.835801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.836615 (+ 814us) service_pool.cc:227] Handling call
0810 23:32:28.873967 (+ 37352us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:28.874142 (+ 175us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:29.970570 (+1096428us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.448885 (+478315us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:30.466946 (+ 18061us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:30.476688 13023 rpcz_store.cc:276] 0810 23:32:28.811868 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.814508 (+ 2640us) service_pool.cc:227] Handling call
0810 23:32:28.881869 (+ 67361us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:28.882286 (+ 417us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:30.013277 (+1130991us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.458505 (+445228us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:30.475053 (+ 16548us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:30.496068 13691 tablet_bootstrap.cc:654] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:30.486516 13700 tablet.cc:1722] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:30.522931 13680 ts_tablet_manager.cc:1918] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:30.533027 13680 log.cc:1196] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/75ead79616d047f9972611623fc943b1
I20260810 23:32:30.544625 13680 ts_tablet_manager.cc:1939] T 75ead79616d047f9972611623fc943b1 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:30.557703 13680 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 28) took 1343 ms. Trace:
I20260810 23:32:30.558187 13680 rpcz_store.cc:276] 0810 23:32:29.214599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:29.237588 (+ 22989us) service_pool.cc:227] Handling call
0810 23:32:29.290380 (+ 52792us) ts_tablet_manager.cc:1145] Deleting tablet 75ead79616d047f9972611623fc943b1
0810 23:32:29.290534 (+ 154us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:30.387214 (+1096680us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.513298 (+126084us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.557602 (+ 44304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":50369,"thread_start_us":32189,"threads_started":1}
I20260810 23:32:30.561158 13665 ts_tablet_manager.cc:1428] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.259s user 0.018s sys 0.007s
I20260810 23:32:30.429378 12806 rpcz_store.cc:276] 0810 23:32:28.837283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.874485 (+ 37202us) service_pool.cc:227] Handling call
0810 23:32:28.876012 (+ 1527us) raft_consensus.cc:1457] Updating replica for 2 ops
0810 23:32:29.265468 (+389456us) raft_consensus.cc:1520] Early marking committed up to index 0
0810 23:32:29.265623 (+ 155us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0810 23:32:29.269648 (+ 4025us) alter_schema_op.cc:126] START. Timestamp: 1786404747944463
0810 23:32:29.722920 (+453272us) log.cc:842] Serialized 199 byte log entry
0810 23:32:29.888739 (+165819us) raft_consensus.cc:1631] Marking committed up to 2
0810 23:32:30.012647 (+123908us) log.cc:842] Serialized 14 byte log entry
0810 23:32:30.426483 (+413836us) raft_consensus.cc:1683] Filling consensus response to leader.
0810 23:32:30.426911 (+ 428us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0810 23:32:30.427259 (+ 348us) raft_consensus.cc:1671] finished
0810 23:32:30.427525 (+ 266us) raft_consensus.cc:1677] UpdateReplicas() finished
0810 23:32:30.428247 (+ 722us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:29.716248 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 23:32:29.760636 (+ 44388us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 23:32:29.761199 (+ 563us) alter_schema_op.cc:65] Acquired schema lock
0810 23:32:29.857505 (+ 96306us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 23:32:30.097789 (+240284us) op_driver.cc:444] REPLICATION: finished
0810 23:32:30.485933 (+388144us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
Metrics: {"mutex_wait_us":19992,"prepare.queue_time_us":123267,"prepare.run_cpu_time_us":13,"prepare.run_wall_time_us":14,"raft.queue_time_us":67468,"raft.run_cpu_time_us":1403,"raft.run_wall_time_us":3443,"thread_start_us":137216,"threads_started":2,"wal-append.queue_time_us":166599,"child_traces":[["op",{"apply.queue_time_us":386657,"prepare.queue_time_us":396616,"prepare.run_cpu_time_us":9073,"prepare.run_wall_time_us":144453,"replication_time_us":827929,"thread_start_us":605427,"threads_started":2}]]}
I20260810 23:32:30.563340 13691 tablet_bootstrap.cc:492] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:30.564103 13691 ts_tablet_manager.cc:1397] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.461s user 0.020s sys 0.030s
W20260810 23:32:30.563894 12454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260810 23:32:30.565007 13665 tablet_bootstrap.cc:492] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:30.571595 13680 tablet_replica.cc:330] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:30.574050 13680 raft_consensus.cc:2229] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:30.574726 13680 raft_consensus.cc:2258] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:30.577546 13701 raft_consensus.cc:482] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:30.567351 13691 raft_consensus.cc:348] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:30.579282 13691 raft_consensus.cc:374] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:30.578558 13701 raft_consensus.cc:504] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:30.579603 13691 raft_consensus.cc:729] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:30.583096 13701 leader_election.cc:288] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:30.583220 13691 consensus_queue.cc:260] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:30.585355 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b01e7b3093472bbeb08e930a6ef1b6" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:30.586195 13059 raft_consensus.cc:2454] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
W20260810 23:32:30.586364 12454 leader_election.cc:334] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:30.586866 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 681674cb24274f10b200f264f155c2d1; no voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:30.586771 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b01e7b3093472bbeb08e930a6ef1b6" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:30.588122 13212 leader_election.cc:302] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:30.581143 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 7be44f795bcd4cdcbdb2f3a03ce5253f (table default.Table_8d417224727445fdb1dc7c3df578e3bc [id=d1a1db95f5f84899b7da1b07aa3d5083]) successfully deleted
W20260810 23:32:30.589656 13213 leader_election.cc:341] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:30.590363 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 26b0c3f3068744e0899c7bb1e2823282 because the tablet was not found. No further retry: Not found: Tablet not found: 26b0c3f3068744e0899c7bb1e2823282
I20260810 23:32:30.591260 13701 raft_consensus.cc:2781] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:30.591729 13701 raft_consensus.cc:482] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:30.592237 13701 raft_consensus.cc:3037] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:30.595958 13691 ts_tablet_manager.cc:1428] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260810 23:32:30.600286 13691 tablet_bootstrap.cc:492] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:30.602246 13680 ts_tablet_manager.cc:1905] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:30.604516 13514 tablet_bootstrap.cc:654] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:30.608233 13665 tablet_bootstrap.cc:654] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:30.610746 12236 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 17) took 1320 ms (client timeout 1000 ms). Trace:
I20260810 23:32:30.620064 13691 tablet_bootstrap.cc:654] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:30.620430 12215 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 80678.3us
I20260810 23:32:30.624946 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b0c3f3068744e0899c7bb1e2823282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:26 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:30.626083 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 75ead79616d047f9972611623fc943b1 (table default.Table_e0603d4692934dce9e07597811ccad0a [id=ad699de357934075aee8d7152f8e130b]) successfully deleted
W20260810 23:32:30.611509 12236 rpcz_store.cc:269] 0810 23:32:29.290879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:29.634176 (+343297us) service_pool.cc:227] Handling call
0810 23:32:30.610627 (+976451us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16761,"reported_tablets":3}
I20260810 23:32:30.650941 13514 tablet_bootstrap.cc:492] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:30.658728 13514 ts_tablet_manager.cc:1397] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.329s user 0.013s sys 0.011s
I20260810 23:32:30.662866 13701 raft_consensus.cc:504] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:30.670279 12237 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 91) took 2006 ms. Trace:
I20260810 23:32:30.674533 13526 ts_tablet_manager.cc:1905] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:30.670982 12237 rpcz_store.cc:276] 0810 23:32:28.663574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:29.287394 (+623820us) service_pool.cc:227] Handling call
0810 23:32:29.620980 (+333586us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7156 00007f0ce6e24034 00007f0ce6e23fc2 00007f0ce6e23ef1 00007f0ce6e23e6c 00007f0ce6e23122 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:29.855895 (+234915us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:30.124633 (+268738us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:30.668405 (+543772us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:30.669237 (+ 832us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:30.670164 (+ 927us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:30.434509 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:30.449920 (+ 15411us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:30.450011 (+ 91us) write_op.cc:435] Acquired schema lock
0810 23:32:30.450072 (+ 61us) tablet.cc:660] Decoding operations
0810 23:32:30.451560 (+ 1488us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:30.451771 (+ 211us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:30.451848 (+ 77us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:30.488968 (+ 37120us) tablet.cc:699] Row locks acquired
0810 23:32:30.489136 (+ 168us) write_op.cc:260] PREPARE: finished
0810 23:32:30.489756 (+ 620us) write_op.cc:270] Start()
0810 23:32:30.490263 (+ 507us) write_op.cc:276] Timestamp: P: 1786404750489632 usec, L: 0
0810 23:32:30.490404 (+ 141us) op_driver.cc:331] REPLICATION: starting
0810 23:32:30.511403 (+ 20999us) log.cc:842] Serialized 432 byte log entry
0810 23:32:30.627548 (+116145us) op_driver.cc:444] REPLICATION: finished
0810 23:32:30.641957 (+ 14409us) write_op.cc:301] APPLY: starting
0810 23:32:30.642052 (+ 95us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:30.642204 (+ 152us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:30.642367 (+ 163us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:30.646778 (+ 4411us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:30.646999 (+ 221us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:30.647060 (+ 61us) write_op.cc:312] APPLY: finished
0810 23:32:30.647397 (+ 337us) log.cc:842] Serialized 28 byte log entry
0810 23:32:30.656382 (+ 8985us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:30.656538 (+ 156us) write_op.cc:454] Released schema lock
0810 23:32:30.656603 (+ 65us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":602744064,"child_traces":[["op",{"apply.queue_time_us":14104,"num_ops":2,"prepare.queue_time_us":285313,"prepare.run_cpu_time_us":12581,"prepare.run_wall_time_us":111077,"raft.queue_time_us":171,"raft.run_cpu_time_us":2357,"raft.run_wall_time_us":2371,"replication_time_us":136945,"thread_start_us":317260,"threads_started":2}]]}
I20260810 23:32:30.678082 12237 master_service.cc:946] ReplaceTablet: received request to replace tablet b9b01e7b3093472bbeb08e930a6ef1b6 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:30.680819 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b01e7b3093472bbeb08e930a6ef1b6" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:30.666355 13665 tablet_bootstrap.cc:492] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:30.682582 13665 ts_tablet_manager.cc:1397] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.118s user 0.018s sys 0.003s
W20260810 23:32:30.688834 13213 leader_election.cc:341] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:30.692993 13691 tablet_bootstrap.cc:492] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:30.693773 13691 ts_tablet_manager.cc:1397] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.094s user 0.015s sys 0.001s
I20260810 23:32:30.696887 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b01e7b3093472bbeb08e930a6ef1b6" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:30.697861 13059 raft_consensus.cc:3037] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:30.700484 13695 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260810 23:32:30.708323 13701 leader_election.cc:288] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:30.709992 13680 ts_tablet_manager.cc:1918] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260810 23:32:30.716858 12226 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 08c45b71432a45269cd6a106fc5f7292 has no consensus state
I20260810 23:32:30.716952 13680 log.cc:1196] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/7be44f795bcd4cdcbdb2f3a03ce5253f
I20260810 23:32:30.717885 12226 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 92) took 2044 ms. Trace:
I20260810 23:32:30.720728 13680 ts_tablet_manager.cc:1939] T 7be44f795bcd4cdcbdb2f3a03ce5253f P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:30.723502 12234 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 93) took 1752 ms. Trace:
I20260810 23:32:30.726320 13665 raft_consensus.cc:348] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:30.727865 13665 raft_consensus.cc:374] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:30.728428 13665 raft_consensus.cc:729] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:30.724224 12234 rpcz_store.cc:276] 0810 23:32:28.970808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:29.866565 (+895757us) service_pool.cc:227] Handling call
0810 23:32:30.064186 (+197621us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:30.112944 (+ 48758us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:30.719884 (+606940us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:30.722685 (+ 2801us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:30.723383 (+ 698us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:30.589447 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:30.590046 (+ 599us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:30.590080 (+ 34us) write_op.cc:435] Acquired schema lock
0810 23:32:30.590106 (+ 26us) tablet.cc:660] Decoding operations
0810 23:32:30.590490 (+ 384us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:30.590540 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:30.590575 (+ 35us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:30.590845 (+ 270us) tablet.cc:699] Row locks acquired
0810 23:32:30.590865 (+ 20us) write_op.cc:260] PREPARE: finished
0810 23:32:30.591071 (+ 206us) write_op.cc:270] Start()
0810 23:32:30.591194 (+ 123us) write_op.cc:276] Timestamp: P: 1786404750591012 usec, L: 0
0810 23:32:30.591216 (+ 22us) op_driver.cc:331] REPLICATION: starting
0810 23:32:30.595364 (+ 4148us) log.cc:842] Serialized 432 byte log entry
0810 23:32:30.710407 (+115043us) op_driver.cc:444] REPLICATION: finished
0810 23:32:30.716307 (+ 5900us) write_op.cc:301] APPLY: starting
0810 23:32:30.716423 (+ 116us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:30.716673 (+ 250us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:30.717070 (+ 397us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:30.717976 (+ 906us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:30.718314 (+ 338us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:30.718405 (+ 91us) write_op.cc:312] APPLY: finished
0810 23:32:30.718838 (+ 433us) log.cc:842] Serialized 28 byte log entry
0810 23:32:30.719201 (+ 363us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:30.719300 (+ 99us) write_op.cc:454] Released schema lock
0810 23:32:30.719440 (+ 140us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:30.719828 (+ 388us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:30.719903 (+ 75us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5662,"num_ops":2,"prepare.queue_time_us":316099,"prepare.run_cpu_time_us":3021,"prepare.run_wall_time_us":7220,"replication_time_us":119061,"threads_started":1}]]}
I20260810 23:32:30.722560 13691 raft_consensus.cc:348] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:30.732052 13691 raft_consensus.cc:374] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:30.732656 13691 raft_consensus.cc:729] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:30.736691 13691 consensus_queue.cc:260] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:30.734580 13680 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 30) took 1435 ms. Trace:
I20260810 23:32:30.733407 13059 raft_consensus.cc:2454] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:30.738941 13680 rpcz_store.cc:276] 0810 23:32:29.300105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:29.306173 (+ 6068us) service_pool.cc:227] Handling call
0810 23:32:30.571344 (+1265171us) ts_tablet_manager.cc:1145] Deleting tablet 7be44f795bcd4cdcbdb2f3a03ce5253f
0810 23:32:30.571406 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:30.625602 (+ 54196us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.709479 (+ 83877us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.734496 (+ 25017us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1234240}
I20260810 23:32:30.741667 13212 leader_election.cc:302] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:30.743395 13701 raft_consensus.cc:2781] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:30.740073 13665 consensus_queue.cc:260] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:30.748308 13680 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 29) took 1452 ms. Trace:
I20260810 23:32:30.749403 13680 rpcz_store.cc:276] 0810 23:32:29.296709 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:29.297024 (+ 315us) service_pool.cc:227] Handling call
0810 23:32:30.747752 (+1450728us) ts_tablet_manager.cc:1145] Deleting tablet 151ba1dd24574739a9f0d33c2257235d
0810 23:32:30.747822 (+ 70us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:30.748197 (+ 375us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1410176}
I20260810 23:32:30.746996 13691 ts_tablet_manager.cc:1428] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.053s user 0.013s sys 0.002s
I20260810 23:32:30.752306 13691 tablet_bootstrap.cc:492] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:30.756011 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_426770680e7f4c9fbefdf0cdcb974327"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:30.758114 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_426770680e7f4c9fbefdf0cdcb974327 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:30.754550 13680 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 31) took 1299 ms. Trace:
I20260810 23:32:30.759385 13680 rpcz_store.cc:276] 0810 23:32:29.479691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:29.551181 (+ 71490us) service_pool.cc:227] Handling call
0810 23:32:30.752928 (+1201747us) ts_tablet_manager.cc:1145] Deleting tablet 585baf44aca142199fbc268ff3bcd8f4
0810 23:32:30.752992 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:30.754477 (+ 1485us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1187499}
I20260810 23:32:30.762174 13680 tablet_replica.cc:330] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:30.763779 13680 raft_consensus.cc:2229] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:30.764716 13680 raft_consensus.cc:2258] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:30.768389 13680 ts_tablet_manager.cc:1905] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:30.769629 13701 raft_consensus.cc:686] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:30.771616 13701 consensus_queue.cc:237] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:30.776634 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 7be44f795bcd4cdcbdb2f3a03ce5253f (table default.Table_8d417224727445fdb1dc7c3df578e3bc [id=d1a1db95f5f84899b7da1b07aa3d5083]) successfully deleted
W20260810 23:32:30.778319 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 151ba1dd24574739a9f0d33c2257235d because the tablet was not found. No further retry: Not found: Tablet not found: 151ba1dd24574739a9f0d33c2257235d
I20260810 23:32:30.778831 13691 tablet_bootstrap.cc:654] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:30.783758 13665 ts_tablet_manager.cc:1428] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.101s user 0.008s sys 0.009s
I20260810 23:32:30.785077 13665 tablet_bootstrap.cc:492] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:30.663661 13514 raft_consensus.cc:348] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:30.803364 13514 raft_consensus.cc:374] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:30.803198 13282 tablet_service.cc:1467] Processing CreateTablet for tablet dd83bbe7a1664c278098fa66b9c6895e (DEFAULT_TABLE table=default.Table_7c223ce60be04a498d08155325785819 [id=0eeab77fa3d54892a91f87f2fbdd673d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:30.803718 13514 raft_consensus.cc:729] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:30.804772 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd83bbe7a1664c278098fa66b9c6895e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:30.809173 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 (DEFAULT_TABLE table=default.Table_e6944762a73d4c80ae0866fad8945b41 [id=50e2197e7c13449caeebf39a42ba347c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:30.810451 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1f7c16b6bc4dc0bac80ac55332f056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:30.813177 12766 tablet_service.cc:1467] Processing CreateTablet for tablet dd83bbe7a1664c278098fa66b9c6895e (DEFAULT_TABLE table=default.Table_7c223ce60be04a498d08155325785819 [id=0eeab77fa3d54892a91f87f2fbdd673d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:30.814693 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd83bbe7a1664c278098fa66b9c6895e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:30.811484 13514 consensus_queue.cc:260] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:30.816996 12767 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0d6cd2ad1d4a65929b0d856a171592 (DEFAULT_TABLE table=default.Table_9e4b92615b33447baec585c92237ff05 [id=287124fe670e4041a492d9a4d4524980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:30.818341 12767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0d6cd2ad1d4a65929b0d856a171592. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:30.820853 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 (DEFAULT_TABLE table=default.Table_e6944762a73d4c80ae0866fad8945b41 [id=50e2197e7c13449caeebf39a42ba347c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:30.822175 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1f7c16b6bc4dc0bac80ac55332f056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:30.822239 13514 ts_tablet_manager.cc:1428] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.163s user 0.009s sys 0.001s
I20260810 23:32:30.823267 13514 tablet_bootstrap.cc:492] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:30.824955 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 08c45b71432a45269cd6a106fc5f7292 (DEFAULT_TABLE table=default.Table_41341e2688304faa88c0ccc969e5faa7 [id=b809029bf6cc48ab917e48bbf2109ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:30.826363 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c45b71432a45269cd6a106fc5f7292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:30.832552 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0d6cd2ad1d4a65929b0d856a171592 (DEFAULT_TABLE table=default.Table_9e4b92615b33447baec585c92237ff05 [id=287124fe670e4041a492d9a4d4524980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:30.834095 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0d6cd2ad1d4a65929b0d856a171592. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:30.844959 12240 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_b4921435057046b6bdca92df0d5284dc" } new_table_name: "default.Table_adf76678e17a494185985a544def0f2d" modify_external_catalogs: true
I20260810 23:32:30.845252 12224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 0c0d6cd2ad1d4a65929b0d856a171592
I20260810 23:32:30.847184 13680 ts_tablet_manager.cc:1918] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:30.849174 12228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 8b1f7c16b6bc4dc0bac80ac55332f056
I20260810 23:32:30.849776 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0d6cd2ad1d4a65929b0d856a171592 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:30.852226 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 0c0d6cd2ad1d4a65929b0d856a171592 because the tablet was not found. No further retry: Not found: Tablet not found: 0c0d6cd2ad1d4a65929b0d856a171592
I20260810 23:32:30.847863 13680 log.cc:1196] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/26b0c3f3068744e0899c7bb1e2823282
I20260810 23:32:30.856967 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:30.862357 12772 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0d6cd2ad1d4a65929b0d856a171592 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:30.863641 12772 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:30.855603 13680 ts_tablet_manager.cc:1939] T 26b0c3f3068744e0899c7bb1e2823282 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
W20260810 23:32:30.868387 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 because the tablet was not found. No further retry: Not found: Tablet not found: 8b1f7c16b6bc4dc0bac80ac55332f056
I20260810 23:32:30.872783 13737 raft_consensus.cc:482] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:30.873325 13737 raft_consensus.cc:504] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:30.877228 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df6b412b60f400b829b6042460aa195" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:30.877215 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df6b412b60f400b829b6042460aa195" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:30.877972 13059 raft_consensus.cc:2454] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:30.879369 13212 leader_election.cc:302] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:30.880540 13740 raft_consensus.cc:2781] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:30.880884 13740 raft_consensus.cc:482] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:30.881345 13740 raft_consensus.cc:3037] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:30.885285 13213 leader_election.cc:341] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:30.885425 12223 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet b9b01e7b3093472bbeb08e930a6ef1b6
I20260810 23:32:30.887432 12223 catalog_manager.cc:6565] ReplaceTablet: tablet b9b01e7b3093472bbeb08e930a6ef1b6 deleted and replaced by tablet 046f1a7983ec40489edeaba4d2595b1c
I20260810 23:32:30.890295 12772 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b01e7b3093472bbeb08e930a6ef1b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 046f1a7983ec40489edeaba4d2595b1c) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:30.891810 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b01e7b3093472bbeb08e930a6ef1b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 046f1a7983ec40489edeaba4d2595b1c) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:30.893118 13679 tablet_replica.cc:330] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:30.900166 13737 leader_election.cc:288] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:32:30.900291 12229 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 12) took 791 ms (client timeout 1000 ms). Trace:
W20260810 23:32:30.900671 12229 rpcz_store.cc:269] 0810 23:32:30.109266 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:30.142131 (+ 32865us) service_pool.cc:227] Handling call
0810 23:32:30.469436 (+327305us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b5400003110. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce0206f11 00007f0ce038a851 00007f0ce038ad92 00007f0ce92827c9 00007f0ce921ac36 00007f0ce91ebc28 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec
0810 23:32:30.532668 (+ 63232us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:30.900202 (+367534us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4,"reported_tablets":3,"spinlock_wait_cycles":509980160}
I20260810 23:32:30.902125 13514 tablet_bootstrap.cc:654] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:30.913556 12772 tablet_service.cc:1514] Processing DeleteTablet for tablet 151ba1dd24574739a9f0d33c2257235d with delete_type TABLET_DATA_DELETED (replaced by tablet fd110664bac34f52bfe922a3a76c6290) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:30.915642 12772 tablet_service.cc:1467] Processing CreateTablet for tablet 79da9583d3524347a9be09e35affaaad (DEFAULT_TABLE table=default.Table_275ea19d22624a12a6b3b2c8e985c4af [id=5b4476e15a7e410599b70f84ab267de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:30.916981 12772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79da9583d3524347a9be09e35affaaad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:30.938194 13679 raft_consensus.cc:2229] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:30.949028 13679 pending_rounds.cc:62] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Trying to abort 1 pending ops.
I20260810 23:32:30.949640 13679 pending_rounds.cc:68] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317113859248189440 op_type: NO_OP noop_request { }
I20260810 23:32:30.950850 13679 raft_consensus.cc:2866] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:32:30.951532 13679 raft_consensus.cc:2258] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:30.956998 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ead79616d047f9972611623fc943b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 840400119d20424ea0f03f2bff550d44) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:30.958931 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:30.718441 12226 rpcz_store.cc:276] 0810 23:32:28.673911 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:29.720140 (+1046229us) service_pool.cc:227] Handling call
0810 23:32:29.766918 (+ 46778us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:30.062408 (+295490us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:30.063876 (+ 1468us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:30.099947 (+ 36071us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:30.715672 (+615725us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:30.717769 (+ 2097us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:30.546557 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:30.549624 (+ 3067us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:30.549664 (+ 40us) write_op.cc:435] Acquired schema lock
0810 23:32:30.549683 (+ 19us) tablet.cc:660] Decoding operations
0810 23:32:30.550629 (+ 946us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:30.550751 (+ 122us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:30.550913 (+ 162us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:30.551637 (+ 724us) tablet.cc:699] Row locks acquired
0810 23:32:30.551653 (+ 16us) write_op.cc:260] PREPARE: finished
0810 23:32:30.551864 (+ 211us) write_op.cc:270] Start()
0810 23:32:30.552041 (+ 177us) write_op.cc:276] Timestamp: P: 1786404750551804 usec, L: 0
0810 23:32:30.552066 (+ 25us) op_driver.cc:331] REPLICATION: starting
0810 23:32:30.553312 (+ 1246us) log.cc:842] Serialized 404 byte log entry
0810 23:32:30.710046 (+156734us) op_driver.cc:444] REPLICATION: finished
0810 23:32:30.713167 (+ 3121us) write_op.cc:301] APPLY: starting
0810 23:32:30.713264 (+ 97us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:30.713423 (+ 159us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:30.713550 (+ 127us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:30.713807 (+ 257us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:30.714238 (+ 431us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:30.714338 (+ 100us) write_op.cc:312] APPLY: finished
0810 23:32:30.714713 (+ 375us) log.cc:842] Serialized 22 byte log entry
0810 23:32:30.715030 (+ 317us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:30.715200 (+ 170us) write_op.cc:454] Released schema lock
0810 23:32:30.715246 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:30.715626 (+ 380us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:30.715714 (+ 88us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2815,"apply.run_cpu_time_us":2730,"apply.run_wall_time_us":2837,"num_ops":1,"prepare.queue_time_us":304191,"prepare.run_cpu_time_us":5666,"prepare.run_wall_time_us":42770,"replication_time_us":157843}]]}
W20260810 23:32:30.779382 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 585baf44aca142199fbc268ff3bcd8f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 585baf44aca142199fbc268ff3bcd8f4 already in progress: creating tablet
I20260810 23:32:30.989327 13665 tablet_bootstrap.cc:654] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:31.006333 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 1f791b2df0524997ab1c53a9e5cb39ca (DEFAULT_TABLE table=default.Table_089a1834ee404319b9d766443f10704f [id=a74d0ca6962e44c5801d194da36bf261]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.008742 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f791b2df0524997ab1c53a9e5cb39ca. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:31.021137 12240 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81e26fe4d42445279f12bb17b2841d08 consensus state has no leader
I20260810 23:32:31.021031 13691 tablet_bootstrap.cc:492] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:31.021895 13691 ts_tablet_manager.cc:1397] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.270s user 0.046s sys 0.057s
I20260810 23:32:31.038933 13514 tablet_bootstrap.cc:492] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:31.039467 13514 ts_tablet_manager.cc:1397] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.216s user 0.011s sys 0.002s
I20260810 23:32:31.043162 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 08c45b71432a45269cd6a106fc5f7292 (DEFAULT_TABLE table=default.Table_41341e2688304faa88c0ccc969e5faa7 [id=b809029bf6cc48ab917e48bbf2109ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.046788 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c45b71432a45269cd6a106fc5f7292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.054044 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 81e26fe4d42445279f12bb17b2841d08 (DEFAULT_TABLE table=default.Table_adf76678e17a494185985a544def0f2d [id=5a377948adc7475c8c98141deaf1c274]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.055598 13526 ts_tablet_manager.cc:1918] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:31.055482 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 79da9583d3524347a9be09e35affaaad (DEFAULT_TABLE table=default.Table_275ea19d22624a12a6b3b2c8e985c4af [id=5b4476e15a7e410599b70f84ab267de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.057773 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e26fe4d42445279f12bb17b2841d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.058264 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79da9583d3524347a9be09e35affaaad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.054908 13740 raft_consensus.cc:504] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.068962 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df6b412b60f400b829b6042460aa195" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:31.043982 13514 raft_consensus.cc:348] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.072853 13514 raft_consensus.cc:374] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:31.073143 13514 raft_consensus.cc:729] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:31.074123 13514 consensus_queue.cc:260] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:32:31.074641 13213 leader_election.cc:341] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:31.078540 13514 ts_tablet_manager.cc:1428] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.037s user 0.001s sys 0.009s
I20260810 23:32:31.079746 13514 tablet_bootstrap.cc:492] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:32:31.047008 13742 consensus_queue.cc:1159] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20260810 23:32:31.056268 13526 log.cc:1196] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/75ead79616d047f9972611623fc943b1
I20260810 23:32:31.089653 13526 ts_tablet_manager.cc:1939] T 75ead79616d047f9972611623fc943b1 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:31.089248 13679 ts_tablet_manager.cc:1905] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:31.111819 13691 raft_consensus.cc:348] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.113039 13691 raft_consensus.cc:374] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:31.113399 13691 raft_consensus.cc:729] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:31.114389 13691 consensus_queue.cc:260] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.117889 13280 tablet_service.cc:1467] Processing CreateTablet for tablet 1f791b2df0524997ab1c53a9e5cb39ca (DEFAULT_TABLE table=default.Table_089a1834ee404319b9d766443f10704f [id=a74d0ca6962e44c5801d194da36bf261]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.120158 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f791b2df0524997ab1c53a9e5cb39ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.125272 13514 tablet_bootstrap.cc:654] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:31.133126 13740 leader_election.cc:288] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:31.164476 13526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 27) took 2011 ms. Trace:
I20260810 23:32:31.165239 13526 rpcz_store.cc:276] 0810 23:32:29.153966 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:29.154563 (+ 597us) service_pool.cc:227] Handling call
0810 23:32:29.938876 (+784313us) ts_tablet_manager.cc:1145] Deleting tablet 75ead79616d047f9972611623fc943b1
0810 23:32:29.939799 (+ 923us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:30.430203 (+490404us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b54000b09d4. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3c5352 00007f3cde7db26c 00007f3cdf3e8cfa 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49 00007f3cdf3ffd11 00007f3cdf3fef3d 0000000000580784
0810 23:32:30.987733 (+557530us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:31.055169 (+ 67436us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:31.164332 (+109163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1096285056,"tablet-delete.queue_time_us":769329}
I20260810 23:32:31.176575 13745 raft_consensus.cc:482] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:31.177534 13745 raft_consensus.cc:504] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:31.180673 13665 tablet_bootstrap.cc:492] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:31.181705 13665 ts_tablet_manager.cc:1397] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.397s user 0.016s sys 0.007s
I20260810 23:32:31.136124 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df6b412b60f400b829b6042460aa195" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:31.184679 13059 raft_consensus.cc:3037] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:31.185657 12764 tablet_service.cc:1467] Processing CreateTablet for tablet fd110664bac34f52bfe922a3a76c6290 (DEFAULT_TABLE table=default.Table_0af7ca97e51c47a29d5dd8867b152066 [id=32144cd75df84cfabce64657802c111a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.187049 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd110664bac34f52bfe922a3a76c6290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.194362 13059 raft_consensus.cc:2454] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:31.194399 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2f8688fc6046d58009800c5e288302 (DEFAULT_TABLE table=default.Table_fa01ae9b566c41d1accafece4db355bb [id=1f49f73dd9d34e968defd3a8ade21d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.195786 13212 leader_election.cc:302] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:31.196725 13745 leader_election.cc:288] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:31.197168 13740 raft_consensus.cc:2781] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:31.198232 13740 raft_consensus.cc:686] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:31.199242 13740 consensus_queue.cc:237] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:32:30.987053 12215 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 207702us
I20260810 23:32:31.200906 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2f8688fc6046d58009800c5e288302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.205694 13023 tablet_service.cc:1467] Processing CreateTablet for tablet dd83bbe7a1664c278098fa66b9c6895e (DEFAULT_TABLE table=default.Table_7c223ce60be04a498d08155325785819 [id=0eeab77fa3d54892a91f87f2fbdd673d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.207978 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd83bbe7a1664c278098fa66b9c6895e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:31.210105 12700 leader_election.cc:341] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Not found: Tablet not found: 7be44f795bcd4cdcbdb2f3a03ce5253f
W20260810 23:32:31.211872 12698 leader_election.cc:341] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 7be44f795bcd4cdcbdb2f3a03ce5253f
I20260810 23:32:31.212726 12698 leader_election.cc:302] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:31.212917 12763 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b0c3f3068744e0899c7bb1e2823282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:26 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:31.215348 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:31.237020 13745 raft_consensus.cc:2726] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:31.238075 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 75ead79616d047f9972611623fc943b1 (table default.Table_e0603d4692934dce9e07597811ccad0a [id=ad699de357934075aee8d7152f8e130b]) successfully deleted
I20260810 23:32:31.244140 13282 tablet_service.cc:1467] Processing CreateTablet for tablet fd110664bac34f52bfe922a3a76c6290 (DEFAULT_TABLE table=default.Table_0af7ca97e51c47a29d5dd8867b152066 [id=32144cd75df84cfabce64657802c111a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.246914 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd110664bac34f52bfe922a3a76c6290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.252332 13514 tablet_bootstrap.cc:492] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:31.216132 13526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 28) took 1857 ms. Trace:
I20260810 23:32:31.257638 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 (DEFAULT_TABLE table=default.Table_e6944762a73d4c80ae0866fad8945b41 [id=50e2197e7c13449caeebf39a42ba347c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.257974 13526 rpcz_store.cc:276] 0810 23:32:29.358782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:29.359176 (+ 394us) service_pool.cc:227] Handling call
0810 23:32:31.215130 (+1855954us) ts_tablet_manager.cc:1145] Deleting tablet 585baf44aca142199fbc268ff3bcd8f4
0810 23:32:31.215262 (+ 132us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:31.215985 (+ 723us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1853686}
I20260810 23:32:31.261518 13021 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0d6cd2ad1d4a65929b0d856a171592 (DEFAULT_TABLE table=default.Table_9e4b92615b33447baec585c92237ff05 [id=287124fe670e4041a492d9a4d4524980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.262336 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 81e26fe4d42445279f12bb17b2841d08 (DEFAULT_TABLE table=default.Table_adf76678e17a494185985a544def0f2d [id=5a377948adc7475c8c98141deaf1c274]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.265421 13026 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0d6cd2ad1d4a65929b0d856a171592 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:31.266928 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:31.270349 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b01e7b3093472bbeb08e930a6ef1b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 046f1a7983ec40489edeaba4d2595b1c) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:31.272413 13026 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:24 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:31.274497 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2f8688fc6046d58009800c5e288302 (DEFAULT_TABLE table=default.Table_fa01ae9b566c41d1accafece4db355bb [id=1f49f73dd9d34e968defd3a8ade21d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.275545 13680 tablet_replica.cc:330] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:31.277248 13680 raft_consensus.cc:2229] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:31.278317 13680 raft_consensus.cc:2258] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:31.280720 13665 raft_consensus.cc:348] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.281642 13026 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ead79616d047f9972611623fc943b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 840400119d20424ea0f03f2bff550d44) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:31.282037 13665 raft_consensus.cc:374] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:31.259860 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1f7c16b6bc4dc0bac80ac55332f056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.282716 13665 raft_consensus.cc:729] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:31.288803 13021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0d6cd2ad1d4a65929b0d856a171592. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.288612 13665 consensus_queue.cc:260] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.292366 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e26fe4d42445279f12bb17b2841d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.301147 13031 tablet_service.cc:1467] Processing CreateTablet for tablet 08c45b71432a45269cd6a106fc5f7292 (DEFAULT_TABLE table=default.Table_41341e2688304faa88c0ccc969e5faa7 [id=b809029bf6cc48ab917e48bbf2109ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.303030 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c45b71432a45269cd6a106fc5f7292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.307138 13691 ts_tablet_manager.cc:1428] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.202s user 0.005s sys 0.008s
I20260810 23:32:31.307343 13526 tablet_replica.cc:330] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:31.308692 13691 tablet_bootstrap.cc:492] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:31.308642 13526 raft_consensus.cc:2229] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:31.309180 13526 raft_consensus.cc:2258] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:31.314231 13026 tablet_service.cc:1467] Processing CreateTablet for tablet 79da9583d3524347a9be09e35affaaad (DEFAULT_TABLE table=default.Table_275ea19d22624a12a6b3b2c8e985c4af [id=5b4476e15a7e410599b70f84ab267de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.316124 13026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79da9583d3524347a9be09e35affaaad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.318755 13526 ts_tablet_manager.cc:1905] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:31.252915 13514 ts_tablet_manager.cc:1397] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.173s user 0.005s sys 0.012s
I20260810 23:32:31.320369 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2f8688fc6046d58009800c5e288302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.324736 13680 ts_tablet_manager.cc:1905] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:31.324514 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 585baf44aca142199fbc268ff3bcd8f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 585baf44aca142199fbc268ff3bcd8f4 already in progress: creating tablet
W20260810 23:32:31.340451 12218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.188:45477, user_credentials={real_user=slave}} blocked reactor thread for 459372us
I20260810 23:32:31.354307 13665 ts_tablet_manager.cc:1428] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.168s user 0.008s sys 0.005s
I20260810 23:32:31.360538 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_fd94688e420b45c2a7f44c6b7db5d7e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:31.364426 13514 raft_consensus.cc:348] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.365404 13514 raft_consensus.cc:374] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:31.365713 13514 raft_consensus.cc:729] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:31.366791 13025 tablet_service.cc:1467] Processing CreateTablet for tablet 1f791b2df0524997ab1c53a9e5cb39ca (DEFAULT_TABLE table=default.Table_089a1834ee404319b9d766443f10704f [id=a74d0ca6962e44c5801d194da36bf261]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.366905 13514 consensus_queue.cc:260] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.370716 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f791b2df0524997ab1c53a9e5cb39ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.370513 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 81e26fe4d42445279f12bb17b2841d08 (DEFAULT_TABLE table=default.Table_adf76678e17a494185985a544def0f2d [id=5a377948adc7475c8c98141deaf1c274]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.371358 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 26b0c3f3068744e0899c7bb1e2823282 (table default.Table_bf3c79e609d24e7cb76242c93decb907 [id=afcd22d5c266477cbc0f2160f852346b]) successfully deleted
I20260810 23:32:31.371652 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_9b23c2d0273b4a1193b49dbf9b79f985"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:31.372520 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e26fe4d42445279f12bb17b2841d08. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:31.372700 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 0c0d6cd2ad1d4a65929b0d856a171592 because the tablet was not found. No further retry: Not found: Tablet not found: 0c0d6cd2ad1d4a65929b0d856a171592
W20260810 23:32:31.373925 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b23c2d0273b4a1193b49dbf9b79f985 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:31.377331 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2f8688fc6046d58009800c5e288302 (DEFAULT_TABLE table=default.Table_fa01ae9b566c41d1accafece4db355bb [id=1f49f73dd9d34e968defd3a8ade21d22]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:31.353852 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 0c0d6cd2ad1d4a65929b0d856a171592 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c0d6cd2ad1d4a65929b0d856a171592 already in progress: creating tablet
W20260810 23:32:31.380885 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b1f7c16b6bc4dc0bac80ac55332f056 already in progress: creating tablet
I20260810 23:32:31.384399 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2f8688fc6046d58009800c5e288302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.384701 13740 raft_consensus.cc:482] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:32:31.385674 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 because the tablet was not found. No further retry: Not found: Tablet not found: 8b1f7c16b6bc4dc0bac80ac55332f056
I20260810 23:32:31.385522 13740 raft_consensus.cc:504] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.391341 13740 leader_election.cc:288] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:31.392697 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c53785fdd24df79f138df6176e75d2" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:31.395788 13665 tablet_bootstrap.cc:492] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:31.397205 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c53785fdd24df79f138df6176e75d2" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:31.398185 13059 raft_consensus.cc:2454] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:31.402755 13018 tablet_service.cc:1467] Processing CreateTablet for tablet fd110664bac34f52bfe922a3a76c6290 (DEFAULT_TABLE table=default.Table_0af7ca97e51c47a29d5dd8867b152066 [id=32144cd75df84cfabce64657802c111a]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:31.400857 13213 leader_election.cc:341] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:31.406447 13691 tablet_bootstrap.cc:654] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:31.408510 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd110664bac34f52bfe922a3a76c6290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:31.407982 13212 leader_election.cc:302] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:31.410360 13740 raft_consensus.cc:2781] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:31.410909 13740 raft_consensus.cc:482] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:31.411319 13740 raft_consensus.cc:3037] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:31.416477 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd94688e420b45c2a7f44c6b7db5d7e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:31.437390 13745 raft_consensus.cc:482] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:31.438167 13745 raft_consensus.cc:504] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:31.421588 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_2f01fd75ce7344b1b5d8068f985516c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:31.445205 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f01fd75ce7344b1b5d8068f985516c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:31.447052 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_5c3b1110e7734eceb6c3530fbdf18113"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:31.453583 12698 leader_election.cc:341] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 26b0c3f3068744e0899c7bb1e2823282
I20260810 23:32:31.448858 12234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_0af7ca97e51c47a29d5dd8867b152066" } modify_external_catalogs: true
I20260810 23:32:31.450174 12237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_275ea19d22624a12a6b3b2c8e985c4af" } modify_external_catalogs: true
I20260810 23:32:31.456132 13745 leader_election.cc:288] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:31.458410 13514 ts_tablet_manager.cc:1428] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.138s user 0.003s sys 0.003s
I20260810 23:32:31.459568 13514 tablet_bootstrap.cc:492] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:31.462074 13679 ts_tablet_manager.cc:1918] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:31.482965 13526 ts_tablet_manager.cc:1918] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:31.485311 13740 raft_consensus.cc:504] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:32:31.488898 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_41341e2688304faa88c0ccc969e5faa7 [id=b809029bf6cc48ab917e48bbf2109ae1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 08c45b71432a45269cd6a106fc5f7292 consensus state has no leader
I20260810 23:32:31.496264 13526 log.cc:1196] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/b9b01e7b3093472bbeb08e930a6ef1b6
I20260810 23:32:31.498874 13665 tablet_bootstrap.cc:654] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:31.500444 13526 ts_tablet_manager.cc:1939] T b9b01e7b3093472bbeb08e930a6ef1b6 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:31.462687 13679 log.cc:1196] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/b9b01e7b3093472bbeb08e930a6ef1b6
I20260810 23:32:31.508507 13679 ts_tablet_manager.cc:1939] T b9b01e7b3093472bbeb08e930a6ef1b6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:31.491097 12237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_275ea19d22624a12a6b3b2c8e985c4af" } modify_external_catalogs: true
I20260810 23:32:31.512374 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c53785fdd24df79f138df6176e75d2" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:31.512990 13526 tablet_replica.cc:330] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:31.513767 13526 raft_consensus.cc:2229] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:31.514248 13526 raft_consensus.cc:2258] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:31.514528 13213 leader_election.cc:341] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:31.492162 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 70337.3us
I20260810 23:32:31.522253 13526 ts_tablet_manager.cc:1905] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:31.524191 13740 leader_election.cc:288] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:31.528057 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet b9b01e7b3093472bbeb08e930a6ef1b6 (table default.Table_21d34a701c494bc580c412d025ddf4dd [id=b5cc65e503f7413fbade0c94e2e74b14]) successfully deleted
I20260810 23:32:31.503360 12234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_0af7ca97e51c47a29d5dd8867b152066" } modify_external_catalogs: true
W20260810 23:32:31.537087 12700 leader_election.cc:341] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Not found: Tablet not found: 26b0c3f3068744e0899c7bb1e2823282
I20260810 23:32:31.537660 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c53785fdd24df79f138df6176e75d2" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:31.537817 12700 leader_election.cc:302] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:31.538661 13059 raft_consensus.cc:3037] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:31.540100 13745 raft_consensus.cc:2726] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:31.507086 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c3b1110e7734eceb6c3530fbdf18113 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:31.549019 13745 raft_consensus.cc:482] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:31.549788 13745 raft_consensus.cc:504] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.561195 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b08ed744b3420f98d241ad96bca6d0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:31.562537 13338 raft_consensus.cc:2454] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:32:31.564733 12698 leader_election.cc:302] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:31.547375 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet b9b01e7b3093472bbeb08e930a6ef1b6 (table default.Table_21d34a701c494bc580c412d025ddf4dd [id=b5cc65e503f7413fbade0c94e2e74b14]) successfully deleted
W20260810 23:32:31.580278 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 75ead79616d047f9972611623fc943b1 because the tablet was not found. No further retry: Not found: Tablet not found: 75ead79616d047f9972611623fc943b1
W20260810 23:32:31.584189 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f because the tablet was not found. No further retry: Not found: Tablet not found: 7be44f795bcd4cdcbdb2f3a03ce5253f
W20260810 23:32:31.591727 12215 consensus_peers.cc:487] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 -> Peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Couldn't send request to peer 198f534261994047a68e1d509bb2fd8b. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260810 23:32:31.609145 12224 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_21d34a701c494bc580c412d025ddf4dd" } new_table_name: "default.Table_ca9eee6f19c34490b6a9cad315a8f740" modify_external_catalogs: true
I20260810 23:32:31.610239 13665 tablet_bootstrap.cc:492] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:31.611281 13665 ts_tablet_manager.cc:1397] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.256s user 0.016s sys 0.019s
I20260810 23:32:31.618645 13665 raft_consensus.cc:348] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.619938 13665 raft_consensus.cc:374] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:31.620385 13665 raft_consensus.cc:729] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:31.628841 13665 consensus_queue.cc:260] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:32:31.641244 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 76668.1us
I20260810 23:32:31.642392 13665 ts_tablet_manager.cc:1428] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.030s user 0.004s sys 0.009s
I20260810 23:32:31.643287 13745 leader_election.cc:288] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:31.644577 13665 tablet_bootstrap.cc:492] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:31.668875 13764 raft_consensus.cc:2781] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:31.669610 13764 raft_consensus.cc:482] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:31.670343 13764 raft_consensus.cc:3037] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:31.674945 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b08ed744b3420f98d241ad96bca6d0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:31.688935 12240 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_fd3d342652da4d5baad3608431df6041" } new_table_name: "default.Table_4f70cb26f31840209e87e54e64d29c58" modify_external_catalogs: true
I20260810 23:32:31.711424 13665 tablet_bootstrap.cc:654] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:31.711987 12700 leader_election.cc:341] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:31.718750 13059 raft_consensus.cc:2454] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:31.721293 13212 leader_election.cc:302] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:31.724324 13740 raft_consensus.cc:2781] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:31.726547 13740 raft_consensus.cc:686] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:31.733194 13665 tablet_bootstrap.cc:492] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:31.734063 13665 ts_tablet_manager.cc:1397] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.017s
I20260810 23:32:31.740906 13740 consensus_queue.cc:237] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.750054 13665 raft_consensus.cc:348] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.752208 13665 raft_consensus.cc:374] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:31.762686 13665 raft_consensus.cc:729] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:31.763963 13665 consensus_queue.cc:260] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.772010 13665 ts_tablet_manager.cc:1428] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.037s user 0.010s sys 0.008s
I20260810 23:32:31.773339 13665 tablet_bootstrap.cc:492] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:31.801733 13691 tablet_bootstrap.cc:492] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:31.807904 13665 tablet_bootstrap.cc:654] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:31.808789 13691 ts_tablet_manager.cc:1397] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.500s user 0.018s sys 0.010s
I20260810 23:32:31.834648 12226 master_service.cc:946] ReplaceTablet: received request to replace tablet 81e26fe4d42445279f12bb17b2841d08 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:31.842981 12236 master_service.cc:946] ReplaceTablet: received request to replace tablet 5df6b412b60f400b829b6042460aa195 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:31.873155 13665 tablet_bootstrap.cc:492] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:31.873870 13665 ts_tablet_manager.cc:1397] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.101s user 0.020s sys 0.019s
I20260810 23:32:31.886298 13514 tablet_bootstrap.cc:654] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:31.879907 13665 raft_consensus.cc:348] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:31.889529 13665 raft_consensus.cc:374] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:31.890013 13665 raft_consensus.cc:729] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:31.900894 13680 ts_tablet_manager.cc:1918] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:31.901631 13680 log.cc:1196] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/b9b01e7b3093472bbeb08e930a6ef1b6
I20260810 23:32:31.902683 13680 ts_tablet_manager.cc:1939] T b9b01e7b3093472bbeb08e930a6ef1b6 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:31.911399 13665 consensus_queue.cc:260] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:31.925132 13665 ts_tablet_manager.cc:1428] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.051s user 0.008s sys 0.008s
I20260810 23:32:31.922878 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet b9b01e7b3093472bbeb08e930a6ef1b6 (table default.Table_21d34a701c494bc580c412d025ddf4dd [id=b5cc65e503f7413fbade0c94e2e74b14]) successfully deleted
W20260810 23:32:31.943606 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 7be44f795bcd4cdcbdb2f3a03ce5253f because the tablet was not found. No further retry: Not found: Tablet not found: 7be44f795bcd4cdcbdb2f3a03ce5253f
W20260810 23:32:31.946215 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 75ead79616d047f9972611623fc943b1 because the tablet was not found. No further retry: Not found: Tablet not found: 75ead79616d047f9972611623fc943b1
I20260810 23:32:31.949083 13665 tablet_bootstrap.cc:492] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:31.963297 13691 raft_consensus.cc:348] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.964780 13691 raft_consensus.cc:374] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:31.965296 13691 raft_consensus.cc:729] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:31.867235 13764 raft_consensus.cc:504] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:31.994009 13526 ts_tablet_manager.cc:1918] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:31.994508 13526 log.cc:1196] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/151ba1dd24574739a9f0d33c2257235d
I20260810 23:32:31.995244 13526 ts_tablet_manager.cc:1939] T 151ba1dd24574739a9f0d33c2257235d P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:32.002748 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_514d038dba3a4fc399b97e7edc254ad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:32.009691 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_514d038dba3a4fc399b97e7edc254ad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:32.029029 13526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 36) took 1116 ms. Trace:
I20260810 23:32:32.029522 13526 rpcz_store.cc:276] 0810 23:32:30.912647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:30.913351 (+ 704us) service_pool.cc:227] Handling call
0810 23:32:31.512759 (+599408us) ts_tablet_manager.cc:1145] Deleting tablet 151ba1dd24574739a9f0d33c2257235d
0810 23:32:31.512817 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:31.521168 (+ 8351us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b280003b1c8. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3ce02ae8b1 00007f3ce02c21ea 00007f3cde72e49f 00007f3cde7db3fb 00007f3cdf3e8cfa 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49 00007f3cdf3ffd11 00007f3cdf3fef3d
0810 23:32:31.869923 (+348755us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:31.993686 (+123763us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:32.028916 (+ 35230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5885696,"tablet-delete.queue_time_us":598684}
I20260810 23:32:32.030035 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b08ed744b3420f98d241ad96bca6d0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:32.031026 13338 raft_consensus.cc:3037] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:32.032014 13764 leader_election.cc:288] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:32.032106 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 151ba1dd24574739a9f0d33c2257235d (table default.Table_0af7ca97e51c47a29d5dd8867b152066 [id=32144cd75df84cfabce64657802c111a]) successfully deleted
W20260810 23:32:32.037055 12240 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dce0ec83ed8943a0b0788505bb75d1df consensus state has no leader
I20260810 23:32:32.044094 13338 raft_consensus.cc:2454] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:32.047093 12698 leader_election.cc:302] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:32.049345 13764 raft_consensus.cc:2781] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:32.051062 13764 raft_consensus.cc:686] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Becoming Leader. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Running, Role: LEADER
I20260810 23:32:32.053011 13764 consensus_queue.cc:237] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.056061 12228 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet e5b08ed744b3420f98d241ad96bca6d0 (table default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]). Expected version 1 got 0
I20260810 23:32:32.056728 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_994c5ce0643d422595ad55d7f893ae9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:32.058893 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_994c5ce0643d422595ad55d7f893ae9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:32.065248 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 18b29cda94584badb1225477ec2bd1bc (DEFAULT_TABLE table=default.Table_b6ab31c2c3cb48fe84b341db2abba9ff [id=74d3cc5da04c4e098ac075abf30a8a72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.067237 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b29cda94584badb1225477ec2bd1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.076834 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 18b29cda94584badb1225477ec2bd1bc (DEFAULT_TABLE table=default.Table_b6ab31c2c3cb48fe84b341db2abba9ff [id=74d3cc5da04c4e098ac075abf30a8a72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.079319 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b29cda94584badb1225477ec2bd1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.083864 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 3d596299c82b4ff1ac151d68b97ff55b (DEFAULT_TABLE table=default.Table_43485ff6f5de42c7be9cf286a18133fb [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.085969 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d596299c82b4ff1ac151d68b97ff55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.086894 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 3d596299c82b4ff1ac151d68b97ff55b (DEFAULT_TABLE table=default.Table_43485ff6f5de42c7be9cf286a18133fb [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.089386 13740 raft_consensus.cc:482] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:32.090135 13740 raft_consensus.cc:504] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:32.092258 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d596299c82b4ff1ac151d68b97ff55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.097715 13740 leader_election.cc:288] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:32.100577 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217a7582902244f9b0ca65be1c20af22" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:32.101317 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 18b29cda94584badb1225477ec2bd1bc (DEFAULT_TABLE table=default.Table_b6ab31c2c3cb48fe84b341db2abba9ff [id=74d3cc5da04c4e098ac075abf30a8a72]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:32.106387 13212 leader_election.cc:341] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:32.114598 12224 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 046f1a7983ec40489edeaba4d2595b1c consensus state has no leader
W20260810 23:32:32.137393 12229 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 13) took 771 ms (client timeout 1000 ms). Trace:
I20260810 23:32:32.138201 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b01e7b3093472bbeb08e930a6ef1b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 046f1a7983ec40489edeaba4d2595b1c) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:32.139129 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_a0d143fe7e1947e6b947eda2d6042925"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:32.138057 13740 consensus_queue.cc:1035] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:32:32.141651 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0d143fe7e1947e6b947eda2d6042925 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:32.142516 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet b9b01e7b3093472bbeb08e930a6ef1b6 because the tablet was not found. No further retry: Not found: Tablet not found: b9b01e7b3093472bbeb08e930a6ef1b6
I20260810 23:32:32.143253 13526 tablet_replica.cc:330] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:32.144061 13526 raft_consensus.cc:2229] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:32.144629 13526 raft_consensus.cc:2258] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:32.146647 13526 ts_tablet_manager.cc:1905] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:32.147704 13025 tablet_service.cc:1467] Processing CreateTablet for tablet b8986e59b2314bd69b9b1de36f1f2ba7 (DEFAULT_TABLE table=default.Table_d318aced7c0540b3b58280f0285ecf87 [id=cefe7b97c1db42ef8712bf3c7a359af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.151144 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8986e59b2314bd69b9b1de36f1f2ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.159781 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b29cda94584badb1225477ec2bd1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.170089 12772 tablet_service.cc:1467] Processing CreateTablet for tablet 3d596299c82b4ff1ac151d68b97ff55b (DEFAULT_TABLE table=default.Table_43485ff6f5de42c7be9cf286a18133fb [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.170730 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b01e7b3093472bbeb08e930a6ef1b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 046f1a7983ec40489edeaba4d2595b1c) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:32.171399 12772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d596299c82b4ff1ac151d68b97ff55b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:32.177749 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet b9b01e7b3093472bbeb08e930a6ef1b6 because the tablet was not found. No further retry: Not found: Tablet not found: b9b01e7b3093472bbeb08e930a6ef1b6
I20260810 23:32:32.185519 12769 tablet_service.cc:1467] Processing CreateTablet for tablet b8986e59b2314bd69b9b1de36f1f2ba7 (DEFAULT_TABLE table=default.Table_d318aced7c0540b3b58280f0285ecf87 [id=cefe7b97c1db42ef8712bf3c7a359af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.187574 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8986e59b2314bd69b9b1de36f1f2ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.188180 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 046f1a7983ec40489edeaba4d2595b1c (DEFAULT_TABLE table=default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.188540 13282 tablet_service.cc:1467] Processing CreateTablet for tablet b8986e59b2314bd69b9b1de36f1f2ba7 (DEFAULT_TABLE table=default.Table_d318aced7c0540b3b58280f0285ecf87 [id=cefe7b97c1db42ef8712bf3c7a359af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.190222 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8986e59b2314bd69b9b1de36f1f2ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.196736 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046f1a7983ec40489edeaba4d2595b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.197679 12223 catalog_manager.cc:5495] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129). New cstate: current_term: 1 leader_uuid: "191d120f4c7f44fc8475305a6d2e75f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } }
W20260810 23:32:32.137996 12229 rpcz_store.cc:269] 0810 23:32:31.366083 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:31.366309 (+ 226us) service_pool.cc:227] Handling call
0810 23:32:32.137306 (+770997us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8157,"reported_tablets":3}
I20260810 23:32:32.202522 12764 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 41) took 1018 ms. Trace:
I20260810 23:32:32.220507 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 046f1a7983ec40489edeaba4d2595b1c (DEFAULT_TABLE table=default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.221681 13031 tablet_service.cc:1467] Processing CreateTablet for tablet 874ca0c3586b41a8a3074d8988813633 (DEFAULT_TABLE table=default.Table_969f8ae489e24f139b305369d629d7af [id=4bb9a6fbe6e644c583a468af86950cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.222628 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046f1a7983ec40489edeaba4d2595b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.223483 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 874ca0c3586b41a8a3074d8988813633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.227128 13514 tablet_bootstrap.cc:492] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:32.227931 13514 ts_tablet_manager.cc:1397] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.769s user 0.040s sys 0.038s
I20260810 23:32:32.230986 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 046f1a7983ec40489edeaba4d2595b1c (DEFAULT_TABLE table=default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:31.966708 13691 consensus_queue.cc:260] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.236522 12223 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet e5b08ed744b3420f98d241ad96bca6d0 (table default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]). Expected version 1 got 0
I20260810 23:32:32.239524 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217a7582902244f9b0ca65be1c20af22" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:32.203946 12764 rpcz_store.cc:276] 0810 23:32:31.183769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:31.184073 (+ 304us) service_pool.cc:227] Handling call
0810 23:32:31.186401 (+ 2328us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:31.186515 (+ 114us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:31.358412 (+171897us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:31.956519 (+598107us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:32.202403 (+245884us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":174788}
I20260810 23:32:32.246031 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b08ed744b3420f98d241ad96bca6d0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
W20260810 23:32:32.248317 13213 consensus_peers.cc:487] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:32.249275 13213 leader_election.cc:341] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:32.249719 13213 leader_election.cc:302] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:32.249761 13514 raft_consensus.cc:348] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.250595 13514 raft_consensus.cc:374] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:32.250542 13740 raft_consensus.cc:2726] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:32.250850 13514 raft_consensus.cc:729] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:32.261669 13691 ts_tablet_manager.cc:1428] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.451s user 0.005s sys 0.006s
I20260810 23:32:32.262907 13691 tablet_bootstrap.cc:492] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:32.246460 13785 raft_consensus.cc:482] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:32.251542 13514 consensus_queue.cc:260] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.267944 13785 raft_consensus.cc:504] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.269388 13062 raft_consensus.cc:3037] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:32.269870 13514 ts_tablet_manager.cc:1428] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.041s user 0.010s sys 0.000s
I20260810 23:32:32.270725 13526 ts_tablet_manager.cc:1918] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:32.271104 13526 log.cc:1196] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/26b0c3f3068744e0899c7bb1e2823282
I20260810 23:32:32.271780 13526 ts_tablet_manager.cc:1939] T 26b0c3f3068744e0899c7bb1e2823282 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:32.279593 13062 raft_consensus.cc:2454] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:32.292673 13526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 42) took 1081 ms. Trace:
I20260810 23:32:32.293128 13526 rpcz_store.cc:276] 0810 23:32:31.210589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:31.211797 (+ 1208us) service_pool.cc:227] Handling call
0810 23:32:32.030899 (+819102us) ts_tablet_manager.cc:1145] Deleting tablet 26b0c3f3068744e0899c7bb1e2823282
0810 23:32:32.035388 (+ 4489us) spinlock_profiling.cc:243] Waited 2.95 ms on lock 0x7b5800001898. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cd72acf11 00007f3cd7430851 00007f3cd7430e82 00007f3ce02f08b9 00007f3cdf3245d8 00007f3cdf3e8389 00007f3cdf3e8983 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49
0810 23:32:32.143057 (+107669us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:32.183943 (+ 40886us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:32.270440 (+ 86497us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:32.288978 (+ 18538us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b5800001898. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cd72acf11 00007f3cd7430851 00007f3cd7430e82 00007f3ce02f08b9 00007f3cdf3245d8 00007f3cdf3e8e42 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49 00007f3cdf3ffd11
0810 23:32:32.291947 (+ 2969us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32493056,"tablet-delete.queue_time_us":817163}
I20260810 23:32:32.294638 13514 tablet_bootstrap.cc:492] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:32.297211 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 26b0c3f3068744e0899c7bb1e2823282 (table default.Table_bf3c79e609d24e7cb76242c93decb907 [id=afcd22d5c266477cbc0f2160f852346b]) successfully deleted
I20260810 23:32:32.303136 13740 raft_consensus.cc:482] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:32.303854 13740 raft_consensus.cc:504] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.305939 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046f1a7983ec40489edeaba4d2595b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.307523 13740 leader_election.cc:288] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:32.294720 13526 tablet_replica.cc:330] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:32.309113 13526 raft_consensus.cc:2229] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:32.309715 13526 raft_consensus.cc:2258] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:32.310714 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585baf44aca142199fbc268ff3bcd8f4" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:32.311271 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 874ca0c3586b41a8a3074d8988813633 (DEFAULT_TABLE table=default.Table_969f8ae489e24f139b305369d629d7af [id=4bb9a6fbe6e644c583a468af86950cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.313491 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 874ca0c3586b41a8a3074d8988813633. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:32.313689 13212 leader_election.cc:341] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:32.315769 13514 tablet_bootstrap.cc:654] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:32.319675 13526 ts_tablet_manager.cc:1905] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:32.322222 13785 leader_election.cc:288] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:32.322630 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b08ed744b3420f98d241ad96bca6d0" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:32.324775 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b08ed744b3420f98d241ad96bca6d0" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:32.326078 13338 raft_consensus.cc:2379] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate 191d120f4c7f44fc8475305a6d2e75f3 in this term.
I20260810 23:32:32.326759 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 874ca0c3586b41a8a3074d8988813633 (DEFAULT_TABLE table=default.Table_969f8ae489e24f139b305369d629d7af [id=4bb9a6fbe6e644c583a468af86950cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.328647 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 874ca0c3586b41a8a3074d8988813633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.331194 12769 tablet_service.cc:1467] Processing CreateTablet for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 (DEFAULT_TABLE table=default.Table_e8c26e9362704b6fb84bba331ef685ea [id=e10ad146e4414a6aa58eb11999d5ed27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.331465 12952 leader_election.cc:302] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:32.332659 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.332608 13665 tablet_bootstrap.cc:700] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent opening tablet: real 0.342s user 0.000s sys 0.004s
I20260810 23:32:32.333606 13665 tablet_bootstrap.cc:654] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:32.334712 13290 tablet_service.cc:1467] Processing CreateTablet for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 (DEFAULT_TABLE table=default.Table_e8c26e9362704b6fb84bba331ef685ea [id=e10ad146e4414a6aa58eb11999d5ed27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.336040 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.343492 13785 raft_consensus.cc:2726] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:32.367549 12772 tablet_service.cc:1467] Processing CreateTablet for tablet 840400119d20424ea0f03f2bff550d44 (DEFAULT_TABLE table=default.Table_e0603d4692934dce9e07597811ccad0a [id=ad699de357934075aee8d7152f8e130b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.374627 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585baf44aca142199fbc268ff3bcd8f4" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
W20260810 23:32:32.375675 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 64030.3us
I20260810 23:32:32.376658 12772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840400119d20424ea0f03f2bff550d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.378225 13026 tablet_service.cc:1467] Processing CreateTablet for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 (DEFAULT_TABLE table=default.Table_e8c26e9362704b6fb84bba331ef685ea [id=e10ad146e4414a6aa58eb11999d5ed27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.379601 13026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.391039 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 840400119d20424ea0f03f2bff550d44 (DEFAULT_TABLE table=default.Table_e0603d4692934dce9e07597811ccad0a [id=ad699de357934075aee8d7152f8e130b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.392518 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840400119d20424ea0f03f2bff550d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.416493 13280 tablet_service.cc:1467] Processing CreateTablet for tablet 840400119d20424ea0f03f2bff550d44 (DEFAULT_TABLE table=default.Table_e0603d4692934dce9e07597811ccad0a [id=ad699de357934075aee8d7152f8e130b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.418918 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840400119d20424ea0f03f2bff550d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.420991 13514 tablet_bootstrap.cc:492] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:32.421579 13514 ts_tablet_manager.cc:1397] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.127s user 0.017s sys 0.006s
I20260810 23:32:32.439740 13514 raft_consensus.cc:348] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.447031 13514 raft_consensus.cc:374] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:32.453226 13514 raft_consensus.cc:729] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
W20260810 23:32:32.456957 13213 leader_election.cc:341] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:32.457792 13213 leader_election.cc:302] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:32.462740 13023 tablet_service.cc:1467] Processing CreateTablet for tablet a0ba75f2c2b54ec1968db8feeacf8383 (DEFAULT_TABLE table=default.Table_426770680e7f4c9fbefdf0cdcb974327 [id=8754da7be72b4eef80daacdcdfde82a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.464718 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0ba75f2c2b54ec1968db8feeacf8383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.469995 12226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 81e26fe4d42445279f12bb17b2841d08
I20260810 23:32:32.473827 12234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet fd110664bac34f52bfe922a3a76c6290
I20260810 23:32:32.474100 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e26fe4d42445279f12bb17b2841d08 with delete_type TABLET_DATA_DELETED (replaced by tablet b261673faa964b65af3576c89e302864) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:32.476516 12226 catalog_manager.cc:6565] ReplaceTablet: tablet 81e26fe4d42445279f12bb17b2841d08 deleted and replaced by tablet b261673faa964b65af3576c89e302864
I20260810 23:32:32.478096 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet fd110664bac34f52bfe922a3a76c6290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:32.477555 12234 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 105) took 1053 ms. Trace:
I20260810 23:32:32.483655 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_529a79b85b6f4c5da1ab44678a564943"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:32.485554 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_529a79b85b6f4c5da1ab44678a564943 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:32.490448 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 046f1a7983ec40489edeaba4d2595b1c consensus state has no leader
I20260810 23:32:32.466035 13514 consensus_queue.cc:260] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.482359 12234 rpcz_store.cc:276] 0810 23:32:31.423673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:31.448177 (+ 24504us) service_pool.cc:227] Handling call
0810 23:32:31.529866 (+ 81689us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:31.531558 (+ 1692us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:31.531914 (+ 356us) catalog_manager.cc:2761] Locking tablets
0810 23:32:32.061189 (+529275us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:32.472914 (+411725us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:32.473380 (+ 466us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:32.473610 (+ 230us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:32.477461 (+ 3851us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:32.072803 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:32.073819 (+ 1016us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:32.073879 (+ 60us) write_op.cc:435] Acquired schema lock
0810 23:32:32.073921 (+ 42us) tablet.cc:660] Decoding operations
0810 23:32:32.074732 (+ 811us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:32.074838 (+ 106us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:32.074894 (+ 56us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:32.075562 (+ 668us) tablet.cc:699] Row locks acquired
0810 23:32:32.075611 (+ 49us) write_op.cc:260] PREPARE: finished
0810 23:32:32.075867 (+ 256us) write_op.cc:270] Start()
0810 23:32:32.076101 (+ 234us) write_op.cc:276] Timestamp: P: 1786404752075817 usec, L: 0
0810 23:32:32.076165 (+ 64us) op_driver.cc:331] REPLICATION: starting
0810 23:32:32.077667 (+ 1502us) log.cc:842] Serialized 764 byte log entry
0810 23:32:32.421043 (+343376us) op_driver.cc:444] REPLICATION: finished
0810 23:32:32.426018 (+ 4975us) write_op.cc:301] APPLY: starting
0810 23:32:32.426180 (+ 162us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:32.426399 (+ 219us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:32.426595 (+ 196us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:32.444231 (+ 17636us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:32.444570 (+ 339us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:32.444619 (+ 49us) write_op.cc:312] APPLY: finished
0810 23:32:32.444970 (+ 351us) log.cc:842] Serialized 28 byte log entry
0810 23:32:32.463845 (+ 18875us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:32.465415 (+ 1570us) write_op.cc:454] Released schema lock
0810 23:32:32.472541 (+ 7126us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:32.476851 (+ 4310us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:32.476891 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4726,"apply.run_cpu_time_us":3310,"apply.run_wall_time_us":51190,"mutex_wait_us":5859,"num_ops":2,"prepare.queue_time_us":4072,"prepare.run_cpu_time_us":5226,"prepare.run_wall_time_us":5223,"replication_time_us":344733}]]}
I20260810 23:32:32.489677 12226 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_fa01ae9b566c41d1accafece4db355bb" } new_table_name: "default.Table_783bf3bd81bb4ec294327f9ed5411afb" modify_external_catalogs: true
I20260810 23:32:32.503609 13021 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e26fe4d42445279f12bb17b2841d08 with delete_type TABLET_DATA_DELETED (replaced by tablet b261673faa964b65af3576c89e302864) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:32.508464 13793 raft_consensus.cc:2726] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:32.509251 12236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 5df6b412b60f400b829b6042460aa195
I20260810 23:32:32.515468 12236 catalog_manager.cc:6565] ReplaceTablet: tablet 5df6b412b60f400b829b6042460aa195 deleted and replaced by tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e
I20260810 23:32:32.513370 13514 ts_tablet_manager.cc:1428] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.091s user 0.016s sys 0.005s
I20260810 23:32:32.517468 12236 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_4f70cb26f31840209e87e54e64d29c58" } new_table_name: "default.Table_ca658a34d2fd4a9a93604028a51d249b" modify_external_catalogs: true
I20260810 23:32:32.530289 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 5df6b412b60f400b829b6042460aa195 with delete_type TABLET_DATA_DELETED (replaced by tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:32.528690 13514 tablet_bootstrap.cc:492] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:32.549228 12237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 79da9583d3524347a9be09e35affaaad
I20260810 23:32:32.552713 12764 tablet_service.cc:1467] Processing CreateTablet for tablet a0ba75f2c2b54ec1968db8feeacf8383 (DEFAULT_TABLE table=default.Table_426770680e7f4c9fbefdf0cdcb974327 [id=8754da7be72b4eef80daacdcdfde82a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.551994 12237 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 106) took 1127 ms. Trace:
I20260810 23:32:32.554414 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0ba75f2c2b54ec1968db8feeacf8383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.553755 12237 rpcz_store.cc:276] 0810 23:32:31.424375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:31.449613 (+ 25238us) service_pool.cc:227] Handling call
0810 23:32:31.511695 (+ 62082us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:31.578050 (+ 66355us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:31.616179 (+ 38129us) catalog_manager.cc:2761] Locking tablets
0810 23:32:32.069139 (+452960us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:32.548472 (+479333us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:32.548838 (+ 366us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:32.549105 (+ 267us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:32.551860 (+ 2755us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:32.106147 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:32.106638 (+ 491us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:32.106698 (+ 60us) write_op.cc:435] Acquired schema lock
0810 23:32:32.106717 (+ 19us) tablet.cc:660] Decoding operations
0810 23:32:32.107110 (+ 393us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:32.107155 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:32.107188 (+ 33us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:32.107569 (+ 381us) tablet.cc:699] Row locks acquired
0810 23:32:32.107589 (+ 20us) write_op.cc:260] PREPARE: finished
0810 23:32:32.107733 (+ 144us) write_op.cc:270] Start()
0810 23:32:32.107830 (+ 97us) write_op.cc:276] Timestamp: P: 1786404752107679 usec, L: 0
0810 23:32:32.107853 (+ 23us) op_driver.cc:331] REPLICATION: starting
0810 23:32:32.108590 (+ 737us) log.cc:842] Serialized 764 byte log entry
0810 23:32:32.445416 (+336826us) op_driver.cc:444] REPLICATION: finished
0810 23:32:32.464565 (+ 19149us) write_op.cc:301] APPLY: starting
0810 23:32:32.464693 (+ 128us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:32.464884 (+ 191us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:32.465069 (+ 185us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:32.480452 (+ 15383us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:32.480787 (+ 335us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:32.480866 (+ 79us) write_op.cc:312] APPLY: finished
0810 23:32:32.481198 (+ 332us) log.cc:842] Serialized 28 byte log entry
0810 23:32:32.496048 (+ 14850us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:32.496503 (+ 455us) write_op.cc:454] Released schema lock
0810 23:32:32.496601 (+ 98us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":18913,"num_ops":2,"prepare.queue_time_us":2694,"prepare.run_cpu_time_us":2695,"prepare.run_wall_time_us":2693,"replication_time_us":337406}]]}
I20260810 23:32:32.560549 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet 79da9583d3524347a9be09e35affaaad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:32.562645 13793 raft_consensus.cc:482] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:32.563179 13793 raft_consensus.cc:504] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.566401 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce0ec83ed8943a0b0788505bb75d1df" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
W20260810 23:32:32.569535 13213 leader_election.cc:341] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:32.571827 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e26fe4d42445279f12bb17b2841d08 with delete_type TABLET_DATA_DELETED (replaced by tablet b261673faa964b65af3576c89e302864) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:32.572203 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet fd110664bac34f52bfe922a3a76c6290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:32.573242 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 5df6b412b60f400b829b6042460aa195 with delete_type TABLET_DATA_DELETED (replaced by tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:32.574065 13784 tablet_replica.cc:330] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:32.574767 13291 tablet_service.cc:1467] Processing CreateTablet for tablet a0ba75f2c2b54ec1968db8feeacf8383 (DEFAULT_TABLE table=default.Table_426770680e7f4c9fbefdf0cdcb974327 [id=8754da7be72b4eef80daacdcdfde82a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:32.575901 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0ba75f2c2b54ec1968db8feeacf8383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:32.577632 13017 tablet_service.cc:1514] Processing DeleteTablet for tablet 5df6b412b60f400b829b6042460aa195 with delete_type TABLET_DATA_DELETED (replaced by tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:32.578586 13793 leader_election.cc:288] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:32.579558 13784 raft_consensus.cc:2229] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
W20260810 23:32:32.580775 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 81e26fe4d42445279f12bb17b2841d08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81e26fe4d42445279f12bb17b2841d08 already in progress: creating tablet
I20260810 23:32:32.582392 13784 raft_consensus.cc:2258] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:32.586402 13784 ts_tablet_manager.cc:1905] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:32.586943 13014 tablet_service.cc:1514] Processing DeleteTablet for tablet 79da9583d3524347a9be09e35affaaad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:32.588014 13680 tablet_replica.cc:330] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:32.588874 13680 raft_consensus.cc:2229] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:32.589437 13680 raft_consensus.cc:2258] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:32.586131 13021 tablet_service.cc:1514] Processing DeleteTablet for tablet fd110664bac34f52bfe922a3a76c6290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:32.597777 13680 ts_tablet_manager.cc:1905] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:32.601019 13526 ts_tablet_manager.cc:1918] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:32.601552 13526 log.cc:1196] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/7be44f795bcd4cdcbdb2f3a03ce5253f
I20260810 23:32:32.602384 13526 ts_tablet_manager.cc:1939] T 7be44f795bcd4cdcbdb2f3a03ce5253f P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:32.605436 13526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 43) took 1394 ms. Trace:
I20260810 23:32:32.605844 13526 rpcz_store.cc:276] 0810 23:32:31.211309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:31.214164 (+ 2855us) service_pool.cc:227] Handling call
0810 23:32:32.294550 (+1080386us) ts_tablet_manager.cc:1145] Deleting tablet 7be44f795bcd4cdcbdb2f3a03ce5253f
0810 23:32:32.294593 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:32.389364 (+ 94771us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:32.600731 (+211367us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:32.605349 (+ 4618us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1075779}
I20260810 23:32:32.607540 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 7be44f795bcd4cdcbdb2f3a03ce5253f (table default.Table_8d417224727445fdb1dc7c3df578e3bc [id=d1a1db95f5f84899b7da1b07aa3d5083]) successfully deleted
W20260810 23:32:32.612972 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 81e26fe4d42445279f12bb17b2841d08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81e26fe4d42445279f12bb17b2841d08 already in progress: creating tablet
I20260810 23:32:32.615237 13514 tablet_bootstrap.cc:654] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:32.617237 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet fd110664bac34f52bfe922a3a76c6290 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd110664bac34f52bfe922a3a76c6290 already in progress: creating tablet
I20260810 23:32:32.622891 12224 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:32.628665 12224 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet e5b08ed744b3420f98d241ad96bca6d0 (table default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]). Expected version 1 got 0
I20260810 23:32:32.627837 13526 tablet_replica.cc:330] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:32.630559 13526 raft_consensus.cc:2229] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:32.633654 13764 raft_consensus.cc:482] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:32.636384 13526 raft_consensus.cc:2258] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:32.636360 13764 raft_consensus.cc:504] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.639814 13526 ts_tablet_manager.cc:1905] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:32.642071 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c53785fdd24df79f138df6176e75d2" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:32.630198 12224 catalog_manager.cc:5495] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:32.646740 12224 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet dce0ec83ed8943a0b0788505bb75d1df (table default.Table_4f70cb26f31840209e87e54e64d29c58 [id=e462d54230344223a457f708501d2d30]). Expected version 1 got 0
I20260810 23:32:32.647210 13764 leader_election.cc:288] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:32.648595 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c53785fdd24df79f138df6176e75d2" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:32.650130 13062 raft_consensus.cc:2379] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 191d120f4c7f44fc8475305a6d2e75f3 in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:32:32.652357 13784 ts_tablet_manager.cc:1918] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:32.654608 12700 leader_election.cc:302] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:32.656677 13764 raft_consensus.cc:3037] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:32.662441 13514 tablet_bootstrap.cc:492] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:32.663013 13514 ts_tablet_manager.cc:1397] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.138s user 0.007s sys 0.011s
W20260810 23:32:32.664628 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 81e26fe4d42445279f12bb17b2841d08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81e26fe4d42445279f12bb17b2841d08 already in progress: creating tablet
I20260810 23:32:32.666229 13514 raft_consensus.cc:348] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.662351 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 79da9583d3524347a9be09e35affaaad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:32.667068 13514 raft_consensus.cc:374] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:32.667335 13514 raft_consensus.cc:729] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
W20260810 23:32:32.667094 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet fd110664bac34f52bfe922a3a76c6290 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd110664bac34f52bfe922a3a76c6290 already in progress: creating tablet
I20260810 23:32:32.667778 13784 log.cc:1196] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/5df6b412b60f400b829b6042460aa195
I20260810 23:32:32.669695 13784 ts_tablet_manager.cc:1939] T 5df6b412b60f400b829b6042460aa195 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:32.671314 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce0ec83ed8943a0b0788505bb75d1df" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:32.675573 13764 raft_consensus.cc:2726] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:32.676133 13514 consensus_queue.cc:260] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.688642 13514 ts_tablet_manager.cc:1428] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.025s user 0.011s sys 0.001s
I20260810 23:32:32.689808 13514 tablet_bootstrap.cc:492] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:32.705026 13526 ts_tablet_manager.cc:1918] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:32.705523 13526 log.cc:1196] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/5df6b412b60f400b829b6042460aa195
I20260810 23:32:32.706271 13526 ts_tablet_manager.cc:1939] T 5df6b412b60f400b829b6042460aa195 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:32.706724 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_7fbbf91a369d4a108ee2b0b6701ad270"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:32.712375 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fbbf91a369d4a108ee2b0b6701ad270 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:32.715924 13514 tablet_bootstrap.cc:654] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:32.718237 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 5df6b412b60f400b829b6042460aa195 (table default.Table_10156b698d054ae8845a67255a9daf3d [id=c48d5d1f759b42f3a5c5a6c64588b30f]) successfully deleted
W20260810 23:32:32.719822 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 79da9583d3524347a9be09e35affaaad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79da9583d3524347a9be09e35affaaad already in progress: creating tablet
W20260810 23:32:32.736725 12226 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d2f8688fc6046d58009800c5e288302 consensus state has no leader
I20260810 23:32:32.749203 13793 consensus_queue.cc:1035] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:32.782281 13514 tablet_bootstrap.cc:492] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:32.782768 13514 ts_tablet_manager.cc:1397] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.093s user 0.007s sys 0.022s
I20260810 23:32:32.822212 12240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_5c3b1110e7734eceb6c3530fbdf18113" } modify_external_catalogs: true
I20260810 23:32:32.822482 13514 raft_consensus.cc:348] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:32.823014 12240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_5c3b1110e7734eceb6c3530fbdf18113" } modify_external_catalogs: true
I20260810 23:32:32.823295 13514 raft_consensus.cc:374] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:32.823717 13514 raft_consensus.cc:729] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:32.824513 13514 consensus_queue.cc:260] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:32.832427 13514 ts_tablet_manager.cc:1428] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.049s user 0.018s sys 0.027s
I20260810 23:32:32.822213 12226 master_service.cc:946] ReplaceTablet: received request to replace tablet a0ba75f2c2b54ec1968db8feeacf8383 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:32.837623 13514 tablet_bootstrap.cc:492] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:32:32.837361 12236 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dce0ec83ed8943a0b0788505bb75d1df consensus state has no leader
I20260810 23:32:32.842363 13793 consensus_queue.cc:1035] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:32:32.845897 13212 leader_election.cc:341] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:32.846745 13212 leader_election.cc:302] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:32.865679 13514 tablet_bootstrap.cc:654] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:32.876133 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 5df6b412b60f400b829b6042460aa195 with delete_type TABLET_DATA_DELETED (replaced by tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e) from {username='slave'} at 127.0.0.1:47546
W20260810 23:32:32.878302 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 5df6b412b60f400b829b6042460aa195 because the tablet was not found. No further retry: Not found: Tablet not found: 5df6b412b60f400b829b6042460aa195
I20260810 23:32:32.886648 12228 master_service.cc:946] ReplaceTablet: received request to replace tablet b8986e59b2314bd69b9b1de36f1f2ba7 from {username='slave'} at 127.0.0.1:34742
W20260810 23:32:32.915467 12224 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 14) took 807 ms (client timeout 1000 ms). Trace:
I20260810 23:32:32.916441 13514 tablet_bootstrap.cc:492] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:32.916960 13514 ts_tablet_manager.cc:1397] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.013s
I20260810 23:32:32.928735 13514 raft_consensus.cc:348] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.929536 13514 raft_consensus.cc:374] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:32.929788 13514 raft_consensus.cc:729] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:32.930758 13514 consensus_queue.cc:260] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:32.933691 13514 ts_tablet_manager.cc:1428] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260810 23:32:32.936540 13514 tablet_bootstrap.cc:492] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:32.919010 12229 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet e5b08ed744b3420f98d241ad96bca6d0 (table default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]). Expected version 1 got 0
I20260810 23:32:32.922802 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_d664d07d116e4f2aa4fc263449eb8dd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:32.951556 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d664d07d116e4f2aa4fc263449eb8dd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:32.952096 12234 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:32.925720 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_addbc3e258944d0cb5d2e7fb11214412"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:32.954200 13514 tablet_bootstrap.cc:654] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:32.954903 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_addbc3e258944d0cb5d2e7fb11214412 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:32.928229 12237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_fd94688e420b45c2a7f44c6b7db5d7e5" } modify_external_catalogs: true
I20260810 23:32:32.956441 12237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_fd94688e420b45c2a7f44c6b7db5d7e5" } modify_external_catalogs: true
I20260810 23:32:32.956854 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 585baf44aca142199fbc268ff3bcd8f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:27 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:32.962386 13665 tablet_bootstrap.cc:492] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:32.962963 13665 ts_tablet_manager.cc:1397] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 1.014s user 0.017s sys 0.036s
I20260810 23:32:32.966197 13665 raft_consensus.cc:348] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:32.967001 13665 raft_consensus.cc:374] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:32.967273 13665 raft_consensus.cc:729] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:32.968281 13665 consensus_queue.cc:260] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:32.971349 13793 raft_consensus.cc:482] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:32.971989 13793 raft_consensus.cc:504] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:32.976883 13793 leader_election.cc:288] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:32.977383 13665 ts_tablet_manager.cc:1428] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20260810 23:32:32.978554 13665 tablet_bootstrap.cc:492] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:32.982930 13514 tablet_bootstrap.cc:492] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:32.983634 13514 ts_tablet_manager.cc:1397] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.004s
I20260810 23:32:32.988107 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd83bbe7a1664c278098fa66b9c6895e" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
W20260810 23:32:32.990737 13213 leader_election.cc:341] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:32.993628 13818 raft_consensus.cc:482] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:32.994331 13818 raft_consensus.cc:504] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.000204 13818 leader_election.cc:288] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:33.002303 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217a7582902244f9b0ca65be1c20af22" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:33.000002 12229 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_43485ff6f5de42c7be9cf286a18133fb" } new_table_name: "default.Table_1841b296cea84a33bb2cc47cfb03f460" modify_external_catalogs: true
W20260810 23:32:32.948832 12224 rpcz_store.cc:269] 0810 23:32:32.107722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:32.115493 (+ 7771us) service_pool.cc:227] Handling call
0810 23:32:32.915372 (+799879us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:32.676612 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:32.677588 (+ 976us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:32.677650 (+ 62us) write_op.cc:435] Acquired schema lock
0810 23:32:32.677662 (+ 12us) tablet.cc:660] Decoding operations
0810 23:32:32.678159 (+ 497us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:32.678274 (+ 115us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:32.678331 (+ 57us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:32.678610 (+ 279us) tablet.cc:699] Row locks acquired
0810 23:32:32.678658 (+ 48us) write_op.cc:260] PREPARE: finished
0810 23:32:32.678984 (+ 326us) write_op.cc:270] Start()
0810 23:32:32.679171 (+ 187us) write_op.cc:276] Timestamp: P: 1786404752678890 usec, L: 0
0810 23:32:32.679191 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:32:32.680403 (+ 1212us) log.cc:842] Serialized 535 byte log entry
0810 23:32:32.841044 (+160641us) op_driver.cc:444] REPLICATION: finished
0810 23:32:32.841658 (+ 614us) write_op.cc:301] APPLY: starting
0810 23:32:32.841759 (+ 101us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:32.841917 (+ 158us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:32.842154 (+ 237us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:32.842425 (+ 271us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:32.843048 (+ 623us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:32.843154 (+ 106us) write_op.cc:312] APPLY: finished
0810 23:32:32.843460 (+ 306us) log.cc:842] Serialized 22 byte log entry
0810 23:32:32.843750 (+ 290us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:32.843882 (+ 132us) write_op.cc:454] Released schema lock
0810 23:32:32.843966 (+ 84us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:32.844367 (+ 401us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:32.844427 (+ 60us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":281,"apply.run_cpu_time_us":2964,"apply.run_wall_time_us":44055,"num_ops":1,"prepare.queue_time_us":4610,"prepare.run_cpu_time_us":4321,"prepare.run_wall_time_us":4662,"raft.queue_time_us":139,"raft.run_cpu_time_us":45,"raft.run_wall_time_us":46,"replication_time_us":161694}]]}
W20260810 23:32:32.835987 13025 consensus_meta.cc:287] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.533s user 0.010s sys 0.000s
W20260810 23:32:32.908798 13032 consensus_meta.cc:287] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.629s user 0.000s sys 0.011s
I20260810 23:32:33.011945 13526 tablet_replica.cc:330] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:33.013105 13526 raft_consensus.cc:2229] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260810 23:32:32.931790 13026 consensus_meta.cc:287] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.542s user 0.011s sys 0.000s
I20260810 23:32:33.013764 13526 raft_consensus.cc:2258] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:33.001322 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217a7582902244f9b0ca65be1c20af22" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:33.003453 13514 raft_consensus.cc:348] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.015578 13514 raft_consensus.cc:374] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:33.015959 13514 raft_consensus.cc:729] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
W20260810 23:32:32.986544 13031 consensus_meta.cc:287] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.600s user 0.013s sys 0.001s
I20260810 23:32:33.016830 13514 consensus_queue.cc:260] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:33.020756 12700 leader_election.cc:341] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:33.021718 13526 ts_tablet_manager.cc:1905] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:33.022329 13514 ts_tablet_manager.cc:1428] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.038s user 0.007s sys 0.003s
I20260810 23:32:33.016974 13691 tablet_bootstrap.cc:700] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b: Time spent opening tablet: real 0.365s user 0.005s sys 0.001s
I20260810 23:32:33.027971 13691 tablet_bootstrap.cc:654] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:33.032356 13338 raft_consensus.cc:2454] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:32:33.033100 12224 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet dce0ec83ed8943a0b0788505bb75d1df (table default.Table_ca658a34d2fd4a9a93604028a51d249b [id=e462d54230344223a457f708501d2d30]). Expected version 2 got 0
I20260810 23:32:33.033967 13514 tablet_bootstrap.cc:492] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:33.041647 12698 leader_election.cc:302] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:32:33.043216 13818 raft_consensus.cc:2781] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:33.043814 13818 raft_consensus.cc:482] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:33.044276 13818 raft_consensus.cc:3037] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:33.049872 13665 tablet_bootstrap.cc:654] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:33.061501 13691 tablet_bootstrap.cc:492] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:33.061758 13514 tablet_bootstrap.cc:654] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:33.062328 13691 ts_tablet_manager.cc:1397] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.800s user 0.028s sys 0.004s
I20260810 23:32:33.055796 13818 raft_consensus.cc:504] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.065882 13691 raft_consensus.cc:348] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.069252 13691 raft_consensus.cc:374] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:33.069810 13691 raft_consensus.cc:729] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:33.070946 13691 consensus_queue.cc:260] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.084612 12240 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 466311c412664cf9b1ed19b1b9bd981f
W20260810 23:32:33.087242 13282 consensus_meta.cc:287] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent flushing consensus metadata: real 0.518s user 0.002s sys 0.005s
I20260810 23:32:33.091306 12224 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_90dfb8b49e7d41fc8267c506f1e3f57e" } new_table_name: "default.Table_bacbea393b664097a0b52af2ce5292de" modify_external_catalogs: true
I20260810 23:32:33.092451 13818 leader_election.cc:288] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:33.093276 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_0d38f0b73fd746b4a487e7b1600eae77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:33.097018 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d38f0b73fd746b4a487e7b1600eae77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:33.098565 12770 consensus_meta.cc:287] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.555s user 0.008s sys 0.002s
I20260810 23:32:33.074479 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217a7582902244f9b0ca65be1c20af22" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:33.112391 13062 raft_consensus.cc:3037] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:33.119331 13818 raft_consensus.cc:482] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:33.120069 13818 raft_consensus.cc:504] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.124624 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c45b71432a45269cd6a106fc5f7292" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:33.126544 12700 leader_election.cc:341] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:33.089128 13665 tablet_bootstrap.cc:492] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:33.093855 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217a7582902244f9b0ca65be1c20af22" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:33.125774 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c45b71432a45269cd6a106fc5f7292" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:33.133566 13665 ts_tablet_manager.cc:1397] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.155s user 0.014s sys 0.004s
I20260810 23:32:33.133595 13338 raft_consensus.cc:3037] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:33.123701 13818 leader_election.cc:288] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:33.137586 13665 raft_consensus.cc:348] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.137678 13526 ts_tablet_manager.cc:1918] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:33.138335 13665 raft_consensus.cc:374] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:33.138412 13526 log.cc:1196] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/585baf44aca142199fbc268ff3bcd8f4
I20260810 23:32:33.138746 13665 raft_consensus.cc:729] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:33.139196 13526 ts_tablet_manager.cc:1939] T 585baf44aca142199fbc268ff3bcd8f4 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:33.139426 13665 consensus_queue.cc:260] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:33.141168 12698 leader_election.cc:341] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:33.142279 13665 ts_tablet_manager.cc:1428] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260810 23:32:33.141846 12698 leader_election.cc:302] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:33.142915 13818 raft_consensus.cc:2726] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:33.143326 13665 tablet_bootstrap.cc:492] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:33.144337 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 585baf44aca142199fbc268ff3bcd8f4 (table default.Table_edb5813389c6488ea72bae6a8373f405 [id=5c3bdcf03c3d43cbbbd2b53dcdc8fa95]) successfully deleted
I20260810 23:32:33.150360 12226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet a0ba75f2c2b54ec1968db8feeacf8383
I20260810 23:32:33.154891 12226 catalog_manager.cc:6565] ReplaceTablet: tablet a0ba75f2c2b54ec1968db8feeacf8383 deleted and replaced by tablet 9c29b11192604ac18c21b7bdc0238fb3
I20260810 23:32:33.157258 13514 tablet_bootstrap.cc:492] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:33.158033 13514 ts_tablet_manager.cc:1397] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.124s user 0.007s sys 0.008s
I20260810 23:32:33.158959 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ba75f2c2b54ec1968db8feeacf8383 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c29b11192604ac18c21b7bdc0238fb3) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:33.159042 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ba75f2c2b54ec1968db8feeacf8383 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c29b11192604ac18c21b7bdc0238fb3) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:33.162810 13514 raft_consensus.cc:348] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.163786 13514 raft_consensus.cc:374] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:33.164073 13514 raft_consensus.cc:729] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:33.165139 13514 consensus_queue.cc:260] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.167891 13818 consensus_queue.cc:1035] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:32:33.171856 13023 consensus_meta.cc:287] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.518s user 0.006s sys 0.007s
I20260810 23:32:33.172580 13514 ts_tablet_manager.cc:1428] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20260810 23:32:33.176863 13514 tablet_bootstrap.cc:492] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:33.182013 12228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet b8986e59b2314bd69b9b1de36f1f2ba7
I20260810 23:32:33.190958 13514 tablet_bootstrap.cc:654] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:33.195616 12228 catalog_manager.cc:6565] ReplaceTablet: tablet b8986e59b2314bd69b9b1de36f1f2ba7 deleted and replaced by tablet 4d697e7533af4fc0a15cb41f3fcce4db
W20260810 23:32:33.197815 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet a0ba75f2c2b54ec1968db8feeacf8383 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0ba75f2c2b54ec1968db8feeacf8383 already in progress: creating tablet
I20260810 23:32:33.166330 13338 raft_consensus.cc:2454] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:33.203033 12698 leader_election.cc:302] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:33.204362 13818 raft_consensus.cc:2781] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:33.205186 13818 raft_consensus.cc:686] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Becoming Leader. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Running, Role: LEADER
I20260810 23:32:33.204734 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet b8986e59b2314bd69b9b1de36f1f2ba7 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d697e7533af4fc0a15cb41f3fcce4db) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:33.208091 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet b8986e59b2314bd69b9b1de36f1f2ba7 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d697e7533af4fc0a15cb41f3fcce4db) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:33.206959 13818 consensus_queue.cc:237] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.225472 13062 raft_consensus.cc:2454] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
W20260810 23:32:33.243832 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet b8986e59b2314bd69b9b1de36f1f2ba7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8986e59b2314bd69b9b1de36f1f2ba7 already in progress: creating tablet
I20260810 23:32:33.249248 13514 tablet_bootstrap.cc:492] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:33.249775 13514 ts_tablet_manager.cc:1397] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.076s user 0.011s sys 0.023s
I20260810 23:32:33.256516 13514 raft_consensus.cc:348] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.257579 13514 raft_consensus.cc:374] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:33.257887 13514 raft_consensus.cc:729] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:33.258760 13514 consensus_queue.cc:260] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.275007 13764 consensus_queue.cc:1035] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:33.277549 13514 ts_tablet_manager.cc:1428] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.027s user 0.007s sys 0.003s
I20260810 23:32:33.278954 13514 tablet_bootstrap.cc:492] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:33.286196 13740 raft_consensus.cc:2726] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:33.287971 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 442097us
I20260810 23:32:33.300505 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 5df6b412b60f400b829b6042460aa195 (table default.Table_10156b698d054ae8845a67255a9daf3d [id=c48d5d1f759b42f3a5c5a6c64588b30f]) successfully deleted
W20260810 23:32:33.305156 12229 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d596299c82b4ff1ac151d68b97ff55b consensus state has no leader
I20260810 23:32:33.305720 13514 tablet_bootstrap.cc:654] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:33.309655 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 79da9583d3524347a9be09e35affaaad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79da9583d3524347a9be09e35affaaad already in progress: creating tablet
I20260810 23:32:33.315207 12236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_b6ab31c2c3cb48fe84b341db2abba9ff" } modify_external_catalogs: true
I20260810 23:32:33.316550 12236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_b6ab31c2c3cb48fe84b341db2abba9ff" } modify_external_catalogs: true
I20260810 23:32:33.344388 13764 raft_consensus.cc:482] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:33.345216 13764 raft_consensus.cc:504] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.348584 13514 tablet_bootstrap.cc:492] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:33.349139 13514 ts_tablet_manager.cc:1397] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.071s user 0.012s sys 0.012s
I20260810 23:32:33.352842 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1f7c16b6bc4dc0bac80ac55332f056" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:33.353382 13514 raft_consensus.cc:348] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.354117 13514 raft_consensus.cc:374] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:33.354372 13514 raft_consensus.cc:729] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:33.354938 13514 consensus_queue.cc:260] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:33.356655 12698 leader_election.cc:341] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:33.357565 13764 leader_election.cc:288] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:33.358120 13514 ts_tablet_manager.cc:1428] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260810 23:32:33.359153 13514 tablet_bootstrap.cc:492] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:33.368919 13514 tablet_bootstrap.cc:654] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:33.377375 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_93dd9072dddd48ec859e82f2c6e0244c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:33.384382 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93dd9072dddd48ec859e82f2c6e0244c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:33.398597 13514 tablet_bootstrap.cc:492] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:33.399115 13514 ts_tablet_manager.cc:1397] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.013s
I20260810 23:32:33.402921 13514 raft_consensus.cc:348] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.403666 13514 raft_consensus.cc:374] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:33.403928 13514 raft_consensus.cc:729] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:33.404718 13514 consensus_queue.cc:260] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.407933 13514 ts_tablet_manager.cc:1428] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.008s user 0.003s sys 0.005s
I20260810 23:32:33.408998 13514 tablet_bootstrap.cc:492] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:33.414685 13818 raft_consensus.cc:482] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:33.415642 13818 raft_consensus.cc:504] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:33.419155 13818 leader_election.cc:288] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:33.420101 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1f7c16b6bc4dc0bac80ac55332f056" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:33.426790 13514 tablet_bootstrap.cc:654] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:33.456771 12700 leader_election.cc:341] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:33.457703 12700 leader_election.cc:302] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:33.459233 13818 raft_consensus.cc:2726] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:33.461999 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce0ec83ed8943a0b0788505bb75d1df" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:33.470314 12228 master_service.cc:946] ReplaceTablet: received request to replace tablet 1f791b2df0524997ab1c53a9e5cb39ca from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:33.484855 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce0ec83ed8943a0b0788505bb75d1df" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:33.486192 13336 raft_consensus.cc:2454] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:32:33.492530 12698 leader_election.cc:302] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:33.495056 13764 raft_consensus.cc:2781] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:33.495823 13764 raft_consensus.cc:482] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:33.496407 13764 raft_consensus.cc:3037] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:33.536854 13514 tablet_bootstrap.cc:492] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:33.536841 13280 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 53) took 1122 ms. Trace:
I20260810 23:32:33.537477 13514 ts_tablet_manager.cc:1397] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.129s user 0.054s sys 0.052s
I20260810 23:32:33.537806 13280 rpcz_store.cc:276] 0810 23:32:32.414529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:32.414812 (+ 283us) service_pool.cc:227] Handling call
0810 23:32:32.417912 (+ 3100us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:32.418112 (+ 200us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:32.734510 (+316398us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:33.307404 (+572894us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:33.536511 (+229107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:33.541180 13514 raft_consensus.cc:348] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.541927 13514 raft_consensus.cc:374] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:33.542196 13514 raft_consensus.cc:729] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:33.542987 13514 consensus_queue.cc:260] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.545955 13514 ts_tablet_manager.cc:1428] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.008s user 0.003s sys 0.006s
I20260810 23:32:33.547153 13514 tablet_bootstrap.cc:492] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:33.558935 13514 tablet_bootstrap.cc:654] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:33.572850 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd83bbe7a1664c278098fa66b9c6895e" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:33.589709 13018 consensus_meta.cc:287] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.501s user 0.011s sys 0.000s
I20260810 23:32:33.594746 13691 ts_tablet_manager.cc:1428] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.532s user 0.015s sys 0.008s
W20260810 23:32:33.595351 13691 ts_tablet_manager.cc:1460] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b: Tablet startup took 532ms
W20260810 23:32:33.595757 13691 ts_tablet_manager.cc:1462] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b: Trace:
0810 23:32:28.893334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:28.893996 (+ 662us) service_pool.cc:227] Handling call
0810 23:32:28.902291 (+ 8295us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:28.902631 (+ 340us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:29.964767 (+1062136us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:30.379416 (+414649us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:30.396171 (+ 16755us) inbound_call.cc:173] Queueing success response
0810 23:32:32.262508 (+1866337us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0810 23:32:33.062758 (+800250us) ts_tablet_manager.cc:1429] Starting tablet replica
0810 23:32:33.063374 (+ 616us) tablet_replica.cc:259] Starting instrumentation
0810 23:32:33.064174 (+ 800us) tablet_replica.cc:278] Starting consensus
0810 23:32:33.231305 (+167131us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54000b59d4. stack: 00007fa86ed56033 00007fa86ed55f30 00007fa86e8d7534 00000000004db2b4 00000000004db249 00007fa875c62471 00007fa875cd44fe 00007fa875cd3079 00007fa8760faa50 00007fa876d07480 00007fa876d1b2d1 00007fa876d1b279 00007fa876d1b209 00007fa876d1b1d1 00007fa876d1a4ad 0000000000580784
Metrics: {"raft.queue_time_us":29385,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":589,"spinlock_wait_cycles":242563328,"tablet-open.queue_time_us":1866858,"thread_start_us":360247,"threads_started":1}
I20260810 23:32:33.601789 13018 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 51) took 1387 ms. Trace:
W20260810 23:32:33.605876 12700 leader_election.cc:341] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:33.608192 13018 rpcz_store.cc:276] 0810 23:32:32.214621 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:32.229586 (+ 14965us) service_pool.cc:227] Handling call
0810 23:32:32.233726 (+ 4140us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:32.233896 (+ 170us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:32.759904 (+526008us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:33.592329 (+832425us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:33.601644 (+ 9315us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:33.611047 13665 tablet_bootstrap.cc:654] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:33.604516 13691 tablet_bootstrap.cc:492] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:33.657100 13691 tablet_bootstrap.cc:654] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:33.619562 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ba75f2c2b54ec1968db8feeacf8383 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c29b11192604ac18c21b7bdc0238fb3) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:33.620373 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet b8986e59b2314bd69b9b1de36f1f2ba7 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d697e7533af4fc0a15cb41f3fcce4db) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:33.635211 13212 leader_election.cc:341] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:33.670071 13212 leader_election.cc:302] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:33.636315 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet 585baf44aca142199fbc268ff3bcd8f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:27 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:33.671854 13866 raft_consensus.cc:2726] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:33.681361 13865 raft_consensus.cc:482] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:33.682693 13865 raft_consensus.cc:504] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:33.684453 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 77227.3us
I20260810 23:32:33.686825 13869 tablet_replica.cc:330] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:33.689700 13869 raft_consensus.cc:2229] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:33.670318 13680 ts_tablet_manager.cc:1918] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:33.692407 13680 log.cc:1196] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/5df6b412b60f400b829b6042460aa195
I20260810 23:32:33.693528 13869 raft_consensus.cc:2258] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:33.696996 13680 ts_tablet_manager.cc:1939] T 5df6b412b60f400b829b6042460aa195 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:33.693718 13665 tablet_bootstrap.cc:492] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:33.698071 13665 ts_tablet_manager.cc:1397] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.555s user 0.010s sys 0.008s
W20260810 23:32:33.709656 12214 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 3d596299c82b4ff1ac151d68b97ff55b (table default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3d596299c82b4ff1ac151d68b97ff55b consensus state has no leader
I20260810 23:32:33.711910 13680 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 58) took 1134 ms. Trace:
W20260810 23:32:33.713886 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet a0ba75f2c2b54ec1968db8feeacf8383 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0ba75f2c2b54ec1968db8feeacf8383 already in progress: creating tablet
W20260810 23:32:33.715358 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet b8986e59b2314bd69b9b1de36f1f2ba7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8986e59b2314bd69b9b1de36f1f2ba7 already in progress: creating tablet
I20260810 23:32:33.727342 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e26fe4d42445279f12bb17b2841d08" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:33.730532 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_e81793ce40c14cfe81a319f4cbd088bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:33.740012 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e81793ce40c14cfe81a319f4cbd088bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:33.719264 13665 raft_consensus.cc:348] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.741878 13665 raft_consensus.cc:374] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:33.735531 12234 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_426770680e7f4c9fbefdf0cdcb974327" } new_table_name: "default.Table_dd2cd30e5e2349de9180be01866dc513" modify_external_catalogs: true
I20260810 23:32:33.742941 13869 ts_tablet_manager.cc:1905] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:33.742221 13665 raft_consensus.cc:729] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:33.749830 13665 consensus_queue.cc:260] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.751358 13691 tablet_bootstrap.cc:492] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
W20260810 23:32:33.748806 12698 leader_election.cc:341] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:33.746577 13865 leader_election.cc:288] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:33.754562 13691 ts_tablet_manager.cc:1397] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.150s user 0.015s sys 0.007s
I20260810 23:32:33.757833 13691 raft_consensus.cc:348] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:33.758834 13691 raft_consensus.cc:374] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:33.759302 13691 raft_consensus.cc:729] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:33.764577 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e26fe4d42445279f12bb17b2841d08" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:33.765218 13514 tablet_bootstrap.cc:492] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:33.765724 13514 ts_tablet_manager.cc:1397] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.219s user 0.039s sys 0.074s
W20260810 23:32:33.767226 12700 leader_election.cc:341] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:33.767803 12700 leader_election.cc:302] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:33.712335 13680 rpcz_store.cc:276] 0810 23:32:32.577112 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:32.577387 (+ 275us) service_pool.cc:227] Handling call
0810 23:32:32.587795 (+ 10408us) ts_tablet_manager.cc:1145] Deleting tablet 5df6b412b60f400b829b6042460aa195
0810 23:32:32.587851 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:32.979339 (+391488us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:33.587710 (+608371us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:33.711819 (+124109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":738}
I20260810 23:32:33.773113 13514 raft_consensus.cc:348] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.773980 13514 raft_consensus.cc:374] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:33.774281 13514 raft_consensus.cc:729] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:33.775051 13514 consensus_queue.cc:260] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.777904 13680 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 59) took 1200 ms. Trace:
I20260810 23:32:33.780630 13680 rpcz_store.cc:276] 0810 23:32:32.577576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:32.584876 (+ 7300us) service_pool.cc:227] Handling call
0810 23:32:33.777218 (+1192342us) ts_tablet_manager.cc:1145] Deleting tablet 79da9583d3524347a9be09e35affaaad
0810 23:32:33.777301 (+ 83us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:33.777811 (+ 510us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1189581}
I20260810 23:32:33.783901 13665 ts_tablet_manager.cc:1428] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.085s user 0.008s sys 0.001s
I20260810 23:32:33.787756 13514 ts_tablet_manager.cc:1428] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20260810 23:32:33.675882 13764 raft_consensus.cc:504] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:33.791260 13680 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 57) took 1285 ms. Trace:
I20260810 23:32:33.791589 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce0ec83ed8943a0b0788505bb75d1df" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:33.792313 13338 raft_consensus.cc:3037] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:33.794595 13680 rpcz_store.cc:276] 0810 23:32:32.506231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:32.506406 (+ 175us) service_pool.cc:227] Handling call
0810 23:32:33.790776 (+1284370us) ts_tablet_manager.cc:1145] Deleting tablet fd110664bac34f52bfe922a3a76c6290
0810 23:32:33.790830 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:33.791187 (+ 357us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197364}
I20260810 23:32:33.797057 13514 tablet_bootstrap.cc:492] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:33.800923 13764 leader_election.cc:288] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:33.804531 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce0ec83ed8943a0b0788505bb75d1df" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:33.803884 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_36d49207cc33452db87dd140a5193dc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:33.806322 13864 raft_consensus.cc:2726] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:33.808420 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36d49207cc33452db87dd140a5193dc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:33.826150 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 68274.8us
I20260810 23:32:33.827221 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 5df6b412b60f400b829b6042460aa195 (table default.Table_10156b698d054ae8845a67255a9daf3d [id=c48d5d1f759b42f3a5c5a6c64588b30f]) successfully deleted
W20260810 23:32:33.828828 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 79da9583d3524347a9be09e35affaaad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79da9583d3524347a9be09e35affaaad already in progress: creating tablet
W20260810 23:32:33.829805 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet fd110664bac34f52bfe922a3a76c6290 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd110664bac34f52bfe922a3a76c6290 already in progress: creating tablet
W20260810 23:32:33.833978 12234 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c29b11192604ac18c21b7bdc0238fb3 has no consensus state
W20260810 23:32:33.836884 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet a0ba75f2c2b54ec1968db8feeacf8383 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0ba75f2c2b54ec1968db8feeacf8383 already in progress: creating tablet
I20260810 23:32:33.837329 12234 master_service.cc:946] ReplaceTablet: received request to replace tablet e5b08ed744b3420f98d241ad96bca6d0 from {username='slave'} at 127.0.0.1:34742
W20260810 23:32:33.837688 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet b8986e59b2314bd69b9b1de36f1f2ba7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8986e59b2314bd69b9b1de36f1f2ba7 already in progress: creating tablet
I20260810 23:32:33.839071 13665 tablet_bootstrap.cc:492] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:33.841233 13514 tablet_bootstrap.cc:654] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:33.850576 13691 consensus_queue.cc:260] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:33.858318 13059 raft_consensus.cc:3037] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:33.882019 13866 raft_consensus.cc:482] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:33.882615 13866 raft_consensus.cc:504] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:33.884756 13866 leader_election.cc:288] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:33.890074 13059 raft_consensus.cc:2454] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:33.906453 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0d6cd2ad1d4a65929b0d856a171592" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:33.907677 12806 raft_consensus.cc:2454] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:33.913851 13213 leader_election.cc:302] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:33.915396 13866 raft_consensus.cc:2781] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:33.915844 13866 raft_consensus.cc:482] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:33.916354 13866 raft_consensus.cc:3037] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:33.917439 13338 raft_consensus.cc:2454] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:33.920732 12698 leader_election.cc:302] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:33.921885 13864 raft_consensus.cc:2781] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:33.922461 13864 raft_consensus.cc:686] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Becoming Leader. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Running, Role: LEADER
I20260810 23:32:33.923682 13864 consensus_queue.cc:237] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:33.930033 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_41341e2688304faa88c0ccc969e5faa7 [id=b809029bf6cc48ab917e48bbf2109ae1]). Expected version 1 got 0
I20260810 23:32:33.934593 13665 tablet_bootstrap.cc:654] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:33.940261 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_34eb8fd429274b5d8af145c70aab107b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:33.955459 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34eb8fd429274b5d8af145c70aab107b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:33.906431 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0d6cd2ad1d4a65929b0d856a171592" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:33.954418 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 081b4e1f53c74f768a4bf401ae22b585 (DEFAULT_TABLE table=default.Table_fd94688e420b45c2a7f44c6b7db5d7e5 [id=cc72e0bbd749430abf4dabf1d737c92d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:33.970937 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081b4e1f53c74f768a4bf401ae22b585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:33.973507 13691 ts_tablet_manager.cc:1428] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.219s user 0.009s sys 0.004s
I20260810 23:32:33.975149 13691 tablet_bootstrap.cc:492] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:33.945693 13764 raft_consensus.cc:482] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:32:33.987165 13212 leader_election.cc:341] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:33.996420 13764 raft_consensus.cc:504] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.001690 13764 leader_election.cc:288] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:34.005493 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79da9583d3524347a9be09e35affaaad" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:34.013726 13514 tablet_bootstrap.cc:492] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:34.014470 13514 ts_tablet_manager.cc:1397] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.218s user 0.019s sys 0.027s
I20260810 23:32:33.951293 12229 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet e5b08ed744b3420f98d241ad96bca6d0 (table default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]). Expected version 1 got 0
I20260810 23:32:34.029112 13514 raft_consensus.cc:348] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.030089 13514 raft_consensus.cc:374] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:34.030429 13514 raft_consensus.cc:729] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
W20260810 23:32:34.007812 12698 leader_election.cc:341] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:34.038002 12240 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 38) took 992 ms (client timeout 1000 ms). Trace:
W20260810 23:32:34.038479 12240 rpcz_store.cc:269] 0810 23:32:33.045587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:33.085472 (+ 39885us) service_pool.cc:227] Handling call
0810 23:32:34.037799 (+952327us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260810 23:32:34.043233 13665 tablet_bootstrap.cc:492] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:34.043783 13665 ts_tablet_manager.cc:1397] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.205s user 0.011s sys 0.010s
I20260810 23:32:34.045534 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 081b4e1f53c74f768a4bf401ae22b585 (DEFAULT_TABLE table=default.Table_fd94688e420b45c2a7f44c6b7db5d7e5 [id=cc72e0bbd749430abf4dabf1d737c92d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.046562 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_023c0d2da11a419f8c2eb9dc2a064cee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:34.047704 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081b4e1f53c74f768a4bf401ae22b585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.050140 13869 ts_tablet_manager.cc:1918] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:34.040975 12223 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_2f01fd75ce7344b1b5d8068f985516c5" } new_table_name: "default.Table_3e3ca6011776476baa1807ba610d0d3d" modify_external_catalogs: true
W20260810 23:32:34.054302 12226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 15) took 974 ms (client timeout 1000 ms). Trace:
I20260810 23:32:34.060462 12764 tablet_service.cc:1467] Processing CreateTablet for tablet f935119e4d9b4accb79db98185c80aca (DEFAULT_TABLE table=default.Table_2f01fd75ce7344b1b5d8068f985516c5 [id=ad09aa494a7b4a3b8ade3a20807cbe0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.061630 13869 log.cc:1196] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/585baf44aca142199fbc268ff3bcd8f4
I20260810 23:32:34.058427 13280 tablet_service.cc:1467] Processing CreateTablet for tablet 081b4e1f53c74f768a4bf401ae22b585 (DEFAULT_TABLE table=default.Table_fd94688e420b45c2a7f44c6b7db5d7e5 [id=cc72e0bbd749430abf4dabf1d737c92d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.062927 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081b4e1f53c74f768a4bf401ae22b585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.065582 13291 tablet_service.cc:1467] Processing CreateTablet for tablet f935119e4d9b4accb79db98185c80aca (DEFAULT_TABLE table=default.Table_2f01fd75ce7344b1b5d8068f985516c5 [id=ad09aa494a7b4a3b8ade3a20807cbe0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.066781 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f935119e4d9b4accb79db98185c80aca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.050861 13665 raft_consensus.cc:348] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.068897 13665 raft_consensus.cc:374] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:34.069212 13665 raft_consensus.cc:729] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:34.070065 13665 consensus_queue.cc:260] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.071772 13869 ts_tablet_manager.cc:1939] T 585baf44aca142199fbc268ff3bcd8f4 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:34.074275 13764 consensus_queue.cc:1035] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:34.083298 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f935119e4d9b4accb79db98185c80aca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.085736 13665 ts_tablet_manager.cc:1428] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.042s user 0.005s sys 0.002s
W20260810 23:32:34.089473 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_023c0d2da11a419f8c2eb9dc2a064cee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:34.054981 12226 rpcz_store.cc:269] 0810 23:32:33.079611 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:33.155626 (+ 76015us) service_pool.cc:227] Handling call
0810 23:32:33.320018 (+164392us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:34.053954 (+733936us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":360670592}
I20260810 23:32:34.110306 13665 tablet_bootstrap.cc:492] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:34.112515 12768 tablet_service.cc:1467] Processing CreateTablet for tablet bd77526f1db4455291fde8a001092a09 (DEFAULT_TABLE table=default.Table_9b23c2d0273b4a1193b49dbf9b79f985 [id=963512beef0d4063b8e6a5738abd1d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.114620 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd77526f1db4455291fde8a001092a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.123857 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 585baf44aca142199fbc268ff3bcd8f4 (table default.Table_edb5813389c6488ea72bae6a8373f405 [id=5c3bdcf03c3d43cbbbd2b53dcdc8fa95]) successfully deleted
I20260810 23:32:34.039527 13514 consensus_queue.cc:260] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.129822 13514 ts_tablet_manager.cc:1428] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.115s user 0.005s sys 0.004s
I20260810 23:32:34.130987 13514 tablet_bootstrap.cc:492] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:34.107545 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79da9583d3524347a9be09e35affaaad" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:34.157721 13868 raft_consensus.cc:482] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:34.168658 13282 tablet_service.cc:1467] Processing CreateTablet for tablet bd77526f1db4455291fde8a001092a09 (DEFAULT_TABLE table=default.Table_9b23c2d0273b4a1193b49dbf9b79f985 [id=963512beef0d4063b8e6a5738abd1d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.169025 13023 tablet_service.cc:1467] Processing CreateTablet for tablet f935119e4d9b4accb79db98185c80aca (DEFAULT_TABLE table=default.Table_2f01fd75ce7344b1b5d8068f985516c5 [id=ad09aa494a7b4a3b8ade3a20807cbe0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.170823 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f935119e4d9b4accb79db98185c80aca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.170734 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd77526f1db4455291fde8a001092a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.159726 13868 raft_consensus.cc:504] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.184288 13868 leader_election.cc:288] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:34.184762 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 124259164adf404c9cbe5f4e0d5e16c0 (DEFAULT_TABLE table=default.Table_994c5ce0643d422595ad55d7f893ae9d [id=69a8ac9bc8ae4c53aefbed949cf8ac95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.186988 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124259164adf404c9cbe5f4e0d5e16c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.191157 13022 tablet_service.cc:1467] Processing CreateTablet for tablet bd77526f1db4455291fde8a001092a09 (DEFAULT_TABLE table=default.Table_9b23c2d0273b4a1193b49dbf9b79f985 [id=963512beef0d4063b8e6a5738abd1d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.197053 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_620c572fc75b44fe8ca892789875c1f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:34.198997 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_620c572fc75b44fe8ca892789875c1f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:34.199949 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd77526f1db4455291fde8a001092a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.200620 12235 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_ca658a34d2fd4a9a93604028a51d249b" } new_table_name: "default.Table_94e24b4b27d047a99df427c453fe1752" modify_external_catalogs: true
I20260810 23:32:34.206763 13025 tablet_service.cc:1467] Processing CreateTablet for tablet 124259164adf404c9cbe5f4e0d5e16c0 (DEFAULT_TABLE table=default.Table_994c5ce0643d422595ad55d7f893ae9d [id=69a8ac9bc8ae4c53aefbed949cf8ac95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.209102 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124259164adf404c9cbe5f4e0d5e16c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.214972 13031 tablet_service.cc:1467] Processing CreateTablet for tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e (DEFAULT_TABLE table=default.Table_10156b698d054ae8845a67255a9daf3d [id=c48d5d1f759b42f3a5c5a6c64588b30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.218261 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.220881 12234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet e5b08ed744b3420f98d241ad96bca6d0
I20260810 23:32:34.225097 12234 catalog_manager.cc:6565] ReplaceTablet: tablet e5b08ed744b3420f98d241ad96bca6d0 deleted and replaced by tablet 4d0d9227d8574de8bd73041f3e8043a0
I20260810 23:32:34.241122 13032 tablet_service.cc:1467] Processing CreateTablet for tablet b261673faa964b65af3576c89e302864 (DEFAULT_TABLE table=default.Table_adf76678e17a494185985a544def0f2d [id=5a377948adc7475c8c98141deaf1c274]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.243217 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b261673faa964b65af3576c89e302864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.238742 13866 raft_consensus.cc:504] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.260871 13026 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b08ed744b3420f98d241ad96bca6d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d0d9227d8574de8bd73041f3e8043a0) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:34.262499 13680 tablet_replica.cc:330] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:34.264113 13680 raft_consensus.cc:2229] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:34.265105 13680 raft_consensus.cc:2258] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:34.266422 12237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 081b4e1f53c74f768a4bf401ae22b585
I20260810 23:32:34.270049 13680 ts_tablet_manager.cc:1905] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:34.279713 12236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 18b29cda94584badb1225477ec2bd1bc
I20260810 23:32:34.280972 13691 tablet_bootstrap.cc:654] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:34.285697 12229 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_a0956ecd557b42e292fde54f05842c47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:34.288522 12237 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 123) took 1361 ms. Trace:
I20260810 23:32:34.288884 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_6e13e5c0a9da4cc5a1b85b4a247f8a3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:34.294561 13866 leader_election.cc:288] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:32:34.296054 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e13e5c0a9da4cc5a1b85b4a247f8a3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:34.297583 12229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0956ecd557b42e292fde54f05842c47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:34.289216 12237 rpcz_store.cc:276] 0810 23:32:32.927598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:32.927754 (+ 156us) service_pool.cc:227] Handling call
0810 23:32:32.957052 (+ 29298us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:32.959284 (+ 2232us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:32.959465 (+ 181us) catalog_manager.cc:2761] Locking tablets
0810 23:32:33.925807 (+966342us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:34.264860 (+339053us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:34.265354 (+ 494us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:34.265522 (+ 168us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:34.288390 (+ 22868us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:34.069451 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:34.069982 (+ 531us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:34.070011 (+ 29us) write_op.cc:435] Acquired schema lock
0810 23:32:34.070029 (+ 18us) tablet.cc:660] Decoding operations
0810 23:32:34.070632 (+ 603us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:34.070835 (+ 203us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:34.070878 (+ 43us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:34.071346 (+ 468us) tablet.cc:699] Row locks acquired
0810 23:32:34.071368 (+ 22us) write_op.cc:260] PREPARE: finished
0810 23:32:34.071574 (+ 206us) write_op.cc:270] Start()
0810 23:32:34.071742 (+ 168us) write_op.cc:276] Timestamp: P: 1786404754070332 usec, L: 0
0810 23:32:34.071774 (+ 32us) op_driver.cc:331] REPLICATION: starting
0810 23:32:34.072520 (+ 746us) log.cc:842] Serialized 733 byte log entry
0810 23:32:34.254317 (+181797us) op_driver.cc:444] REPLICATION: finished
0810 23:32:34.261047 (+ 6730us) write_op.cc:301] APPLY: starting
0810 23:32:34.261197 (+ 150us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:34.261543 (+ 346us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:34.261711 (+ 168us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:34.262343 (+ 632us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:34.262829 (+ 486us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:34.262967 (+ 138us) write_op.cc:312] APPLY: finished
0810 23:32:34.263297 (+ 330us) log.cc:842] Serialized 28 byte log entry
0810 23:32:34.264329 (+ 1032us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:34.264481 (+ 152us) write_op.cc:454] Released schema lock
0810 23:32:34.264537 (+ 56us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:34.271719 (+ 7182us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:34.271774 (+ 55us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":16623,"child_traces":[["op",{"apply.queue_time_us":6475,"apply.run_cpu_time_us":3589,"apply.run_wall_time_us":11105,"mutex_wait_us":26954,"num_ops":2,"prepare.queue_time_us":27165,"prepare.run_cpu_time_us":3436,"prepare.run_wall_time_us":8148,"raft.queue_time_us":2754,"raft.run_cpu_time_us":20,"raft.run_wall_time_us":22,"replication_time_us":182372}]]}
I20260810 23:32:34.283777 13665 tablet_bootstrap.cc:654] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:34.333248 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:34.335332 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 88c53785fdd24df79f138df6176e75d2 (table default.Table_bacbea393b664097a0b52af2ce5292de [id=71c9424f327040a68889fc257935c535]). Expected version 1 got 0
I20260810 23:32:34.343638 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet 081b4e1f53c74f768a4bf401ae22b585 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:32 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:34.344333 13026 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b29cda94584badb1225477ec2bd1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:33 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:34.347275 13691 tablet_bootstrap.cc:492] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:34.348140 13691 ts_tablet_manager.cc:1397] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.373s user 0.018s sys 0.019s
I20260810 23:32:34.350700 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1f7c16b6bc4dc0bac80ac55332f056" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:34.346851 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet dce0ec83ed8943a0b0788505bb75d1df (table default.Table_ca658a34d2fd4a9a93604028a51d249b [id=e462d54230344223a457f708501d2d30]). Expected version 2 got 0
I20260810 23:32:34.366497 13514 tablet_bootstrap.cc:654] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:34.361159 13280 tablet_service.cc:1467] Processing CreateTablet for tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e (DEFAULT_TABLE table=default.Table_10156b698d054ae8845a67255a9daf3d [id=c48d5d1f759b42f3a5c5a6c64588b30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.374176 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.376847 12228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 1f791b2df0524997ab1c53a9e5cb39ca
I20260810 23:32:34.378779 12228 catalog_manager.cc:6565] ReplaceTablet: tablet 1f791b2df0524997ab1c53a9e5cb39ca deleted and replaced by tablet c892e6bc117a4fff917342b8eefaf74d
I20260810 23:32:34.389564 13291 tablet_service.cc:1467] Processing CreateTablet for tablet b261673faa964b65af3576c89e302864 (DEFAULT_TABLE table=default.Table_adf76678e17a494185985a544def0f2d [id=5a377948adc7475c8c98141deaf1c274]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.390239 13691 raft_consensus.cc:348] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:34.391134 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b261673faa964b65af3576c89e302864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.391185 13691 raft_consensus.cc:374] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:34.391443 13691 raft_consensus.cc:729] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:34.399513 13026 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f791b2df0524997ab1c53a9e5cb39ca with delete_type TABLET_DATA_DELETED (replaced by tablet c892e6bc117a4fff917342b8eefaf74d) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:34.399098 13691 consensus_queue.cc:260] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:34.402876 12226 catalog_manager.cc:5495] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129). New cstate: current_term: 1 leader_uuid: "191d120f4c7f44fc8475305a6d2e75f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:34.405400 12226 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:34.406790 12226 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 88c53785fdd24df79f138df6176e75d2 (table default.Table_bacbea393b664097a0b52af2ce5292de [id=71c9424f327040a68889fc257935c535]). Expected version 1 got 0
I20260810 23:32:34.407963 12226 catalog_manager.cc:5495] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129). New cstate: current_term: 1 leader_uuid: "191d120f4c7f44fc8475305a6d2e75f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:34.409906 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b08ed744b3420f98d241ad96bca6d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d0d9227d8574de8bd73041f3e8043a0) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:34.410847 13869 tablet_replica.cc:330] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:34.411813 13869 raft_consensus.cc:2229] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:34.412431 13869 raft_consensus.cc:2258] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:34.413970 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 081b4e1f53c74f768a4bf401ae22b585 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:32 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:34.416566 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b29cda94584badb1225477ec2bd1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:33 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:34.417289 12226 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet dce0ec83ed8943a0b0788505bb75d1df (table default.Table_ca658a34d2fd4a9a93604028a51d249b [id=e462d54230344223a457f708501d2d30]). Expected version 2 got 0
I20260810 23:32:34.418210 13869 ts_tablet_manager.cc:1905] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:34.419324 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f791b2df0524997ab1c53a9e5cb39ca with delete_type TABLET_DATA_DELETED (replaced by tablet c892e6bc117a4fff917342b8eefaf74d) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:34.423076 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0d6cd2ad1d4a65929b0d856a171592" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:34.452529 13866 raft_consensus.cc:482] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:34.453146 13866 raft_consensus.cc:504] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.454205 13680 ts_tablet_manager.cc:1918] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:34.454591 13680 log.cc:1196] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/e5b08ed744b3420f98d241ad96bca6d0
I20260810 23:32:34.455353 13680 ts_tablet_manager.cc:1939] T e5b08ed744b3420f98d241ad96bca6d0 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:34.455498 13866 leader_election.cc:288] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:34.461030 13665 tablet_bootstrap.cc:492] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:34.461663 13665 ts_tablet_manager.cc:1397] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.352s user 0.013s sys 0.013s
I20260810 23:32:34.467952 13665 raft_consensus.cc:348] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.476240 13665 raft_consensus.cc:374] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:34.477259 13665 raft_consensus.cc:729] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:34.485008 13665 consensus_queue.cc:260] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.494596 12224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_7fbbf91a369d4a108ee2b0b6701ad270" } modify_external_catalogs: true
I20260810 23:32:34.495337 12224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_7fbbf91a369d4a108ee2b0b6701ad270" } modify_external_catalogs: true
I20260810 23:32:34.450357 13691 ts_tablet_manager.cc:1428] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.101s user 0.006s sys 0.002s
I20260810 23:32:34.506635 13846 raft_consensus.cc:482] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:34.507201 13846 raft_consensus.cc:504] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:34.509773 13846 leader_election.cc:288] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
W20260810 23:32:34.512086 12952 leader_election.cc:341] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 585baf44aca142199fbc268ff3bcd8f4
I20260810 23:32:34.518631 13691 tablet_bootstrap.cc:492] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:34.518811 12228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_089a1834ee404319b9d766443f10704f" } modify_external_catalogs: true
I20260810 23:32:34.519487 12228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_089a1834ee404319b9d766443f10704f" } modify_external_catalogs: true
W20260810 23:32:34.471707 13212 leader_election.cc:341] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:34.520915 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 49281.2us
W20260810 23:32:34.521646 13212 leader_election.cc:341] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:34.523304 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79da9583d3524347a9be09e35affaaad" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:34.518695 13869 ts_tablet_manager.cc:1918] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:34.524362 13869 log.cc:1196] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/e5b08ed744b3420f98d241ad96bca6d0
I20260810 23:32:34.525583 13869 ts_tablet_manager.cc:1939] T e5b08ed744b3420f98d241ad96bca6d0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
W20260810 23:32:34.534888 13212 leader_election.cc:341] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:34.543654 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet e5b08ed744b3420f98d241ad96bca6d0 (table default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]) successfully deleted
I20260810 23:32:34.547209 13665 ts_tablet_manager.cc:1428] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.085s user 0.005s sys 0.003s
I20260810 23:32:34.550007 12772 tablet_service.cc:1467] Processing CreateTablet for tablet 124259164adf404c9cbe5f4e0d5e16c0 (DEFAULT_TABLE table=default.Table_994c5ce0643d422595ad55d7f893ae9d [id=69a8ac9bc8ae4c53aefbed949cf8ac95]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:34.552661 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 081b4e1f53c74f768a4bf401ae22b585 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 081b4e1f53c74f768a4bf401ae22b585 already in progress: creating tablet
I20260810 23:32:34.554133 12772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124259164adf404c9cbe5f4e0d5e16c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.560350 13906 tablet.cc:1722] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:34.564100 13916 tablet.cc:1722] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:34.572974 13665 tablet_bootstrap.cc:492] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:34.572521 13764 raft_consensus.cc:482] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:34.573450 13764 raft_consensus.cc:504] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.576750 13764 leader_election.cc:288] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
W20260810 23:32:34.577666 12215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111) [suppressed 4 similar messages]
I20260810 23:32:34.578547 13864 raft_consensus.cc:482] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:34.579042 13864 raft_consensus.cc:504] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:34.579573 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 18b29cda94584badb1225477ec2bd1bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18b29cda94584badb1225477ec2bd1bc already in progress: creating tablet
W20260810 23:32:34.580914 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 1f791b2df0524997ab1c53a9e5cb39ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f791b2df0524997ab1c53a9e5cb39ca already in progress: creating tablet
I20260810 23:32:34.581041 13864 leader_election.cc:288] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:34.581473 13864 raft_consensus.cc:482] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:34.581858 13864 raft_consensus.cc:504] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.583676 13864 leader_election.cc:288] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:34.607617 13691 tablet_bootstrap.cc:654] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:34.623324 13665 tablet_bootstrap.cc:654] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:34.639614 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 61964.5us
I20260810 23:32:34.641487 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet e5b08ed744b3420f98d241ad96bca6d0 (table default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]) successfully deleted
W20260810 23:32:34.642325 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 081b4e1f53c74f768a4bf401ae22b585 because the tablet was not found. No further retry: Not found: Tablet not found: 081b4e1f53c74f768a4bf401ae22b585
W20260810 23:32:34.643079 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 18b29cda94584badb1225477ec2bd1bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18b29cda94584badb1225477ec2bd1bc already in progress: creating tablet
W20260810 23:32:34.644119 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 1f791b2df0524997ab1c53a9e5cb39ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f791b2df0524997ab1c53a9e5cb39ca already in progress: creating tablet
I20260810 23:32:34.660447 13691 tablet_bootstrap.cc:492] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
W20260810 23:32:34.661329 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 193183us
I20260810 23:32:34.663434 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f791b2df0524997ab1c53a9e5cb39ca" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:34.664217 13338 raft_consensus.cc:2454] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:32:34.666494 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd83bbe7a1664c278098fa66b9c6895e" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:34.667090 13338 raft_consensus.cc:2454] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:32:34.667768 13691 ts_tablet_manager.cc:1397] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.149s user 0.011s sys 0.003s
I20260810 23:32:34.671726 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2f8688fc6046d58009800c5e288302" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:34.675112 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1f7c16b6bc4dc0bac80ac55332f056" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:34.676311 12806 raft_consensus.cc:2454] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:34.678014 13665 tablet_bootstrap.cc:492] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:34.678577 13665 ts_tablet_manager.cc:1397] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.106s user 0.007s sys 0.007s
I20260810 23:32:34.678311 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0d6cd2ad1d4a65929b0d856a171592" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:34.679316 12806 raft_consensus.cc:3037] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:34.683125 13691 raft_consensus.cc:348] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.684005 13691 raft_consensus.cc:374] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:34.684362 13691 raft_consensus.cc:729] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:34.685338 13691 consensus_queue.cc:260] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.690212 13691 ts_tablet_manager.cc:1428] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260810 23:32:34.692530 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79da9583d3524347a9be09e35affaaad" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:34.693519 12805 raft_consensus.cc:2454] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:34.704152 13691 tablet_bootstrap.cc:492] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:34.705027 12698 leader_election.cc:302] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:34.706035 12768 tablet_service.cc:1467] Processing CreateTablet for tablet b261673faa964b65af3576c89e302864 (DEFAULT_TABLE table=default.Table_adf76678e17a494185985a544def0f2d [id=5a377948adc7475c8c98141deaf1c274]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:34.692662 12700 leader_election.cc:341] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:34.707664 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b261673faa964b65af3576c89e302864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.707542 12700 leader_election.cc:302] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:32:34.713326 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 217a7582902244f9b0ca65be1c20af22 consensus state has no leader
I20260810 23:32:34.714893 13909 raft_consensus.cc:2781] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:34.715730 13909 raft_consensus.cc:482] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:34.687131 13665 raft_consensus.cc:348] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.716229 13665 raft_consensus.cc:374] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:34.716156 13909 raft_consensus.cc:3037] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:34.716532 13665 raft_consensus.cc:729] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:34.717219 13514 tablet_bootstrap.cc:492] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:34.718010 13514 ts_tablet_manager.cc:1397] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.587s user 0.015s sys 0.010s
I20260810 23:32:34.718921 12770 tablet_service.cc:1467] Processing CreateTablet for tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e (DEFAULT_TABLE table=default.Table_10156b698d054ae8845a67255a9daf3d [id=c48d5d1f759b42f3a5c5a6c64588b30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:34.722169 13665 consensus_queue.cc:260] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.730334 13691 tablet_bootstrap.cc:654] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:34.740281 12223 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f935119e4d9b4accb79db98185c80aca consensus state has no leader
I20260810 23:32:34.748147 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_f34283baaf0d495b8a5b076e45292979"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:34.754037 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f34283baaf0d495b8a5b076e45292979 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:34.754387 13665 ts_tablet_manager.cc:1428] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.075s user 0.003s sys 0.003s
I20260810 23:32:34.756060 13691 tablet_bootstrap.cc:492] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:34.723939 13864 raft_consensus.cc:2726] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:34.756742 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 64966.1us
I20260810 23:32:34.759212 13691 ts_tablet_manager.cc:1397] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.013s
I20260810 23:32:34.762982 13691 raft_consensus.cc:348] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.763811 13691 raft_consensus.cc:374] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:34.764108 13691 raft_consensus.cc:729] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:34.770578 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b29cda94584badb1225477ec2bd1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:33 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:34.772934 12766 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f791b2df0524997ab1c53a9e5cb39ca with delete_type TABLET_DATA_DELETED (replaced by tablet c892e6bc117a4fff917342b8eefaf74d) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:34.774055 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:34.773729 13514 raft_consensus.cc:348] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.774463 13514 raft_consensus.cc:374] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:34.774768 13514 raft_consensus.cc:729] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:34.775454 13514 consensus_queue.cc:260] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.721562 12698 leader_election.cc:302] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:34.756758 12772 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b08ed744b3420f98d241ad96bca6d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d0d9227d8574de8bd73041f3e8043a0) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:34.777215 13665 tablet_bootstrap.cc:492] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:34.777954 12769 tablet_service.cc:1514] Processing DeleteTablet for tablet 081b4e1f53c74f768a4bf401ae22b585 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:32 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:34.780489 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f791b2df0524997ab1c53a9e5cb39ca" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:34.783319 13665 tablet_bootstrap.cc:654] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:34.786170 13691 consensus_queue.cc:260] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.788484 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2f8688fc6046d58009800c5e288302" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:34.787810 13846 raft_consensus.cc:482] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:34.789667 13846 raft_consensus.cc:504] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.790868 13935 tablet_replica.cc:330] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:34.791743 13846 leader_election.cc:288] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:34.792999 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd83bbe7a1664c278098fa66b9c6895e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:34.793752 13338 raft_consensus.cc:2454] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:34.795423 13864 raft_consensus.cc:2781] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260810 23:32:34.795349 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 73974.9us
I20260810 23:32:34.795845 13864 raft_consensus.cc:482] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:34.796190 13864 raft_consensus.cc:3037] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:34.796434 12698 leader_election.cc:341] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:34.799711 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 18b29cda94584badb1225477ec2bd1bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18b29cda94584badb1225477ec2bd1bc already in progress: creating tablet
I20260810 23:32:34.804936 13864 raft_consensus.cc:504] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.812505 13213 leader_election.cc:302] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:32:34.813943 13213 leader_election.cc:302] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:32:34.814745 13866 raft_consensus.cc:2781] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:34.815157 13866 raft_consensus.cc:482] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:34.815477 13866 raft_consensus.cc:3037] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:34.824635 13868 raft_consensus.cc:2781] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:34.825320 13868 raft_consensus.cc:482] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:34.825845 13868 raft_consensus.cc:3037] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:34.836769 13514 ts_tablet_manager.cc:1428] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.118s user 0.000s sys 0.010s
I20260810 23:32:34.838260 13514 tablet_bootstrap.cc:492] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:34.838651 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd83bbe7a1664c278098fa66b9c6895e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:34.839874 12805 raft_consensus.cc:2379] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate 191d120f4c7f44fc8475305a6d2e75f3 in this term.
I20260810 23:32:34.841905 13864 leader_election.cc:288] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:34.795037 12952 leader_election.cc:302] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
W20260810 23:32:34.851888 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 56958.6us
I20260810 23:32:34.852320 13846 raft_consensus.cc:2781] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:34.801064 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd83bbe7a1664c278098fa66b9c6895e" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:34.852952 13846 raft_consensus.cc:482] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:34.806725 13691 ts_tablet_manager.cc:1428] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.047s user 0.005s sys 0.005s
I20260810 23:32:34.853546 13846 raft_consensus.cc:3037] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:34.854676 13691 tablet_bootstrap.cc:492] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
W20260810 23:32:34.861442 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 47541.1us
I20260810 23:32:34.872727 12806 raft_consensus.cc:2454] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:34.870924 13691 tablet_bootstrap.cc:654] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:34.880479 12958 leader_election.cc:341] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 585baf44aca142199fbc268ff3bcd8f4
I20260810 23:32:34.881177 12958 leader_election.cc:302] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:34.882984 13936 raft_consensus.cc:2726] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:34.890971 13665 tablet_bootstrap.cc:492] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:34.891616 13665 ts_tablet_manager.cc:1397] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.115s user 0.006s sys 0.006s
I20260810 23:32:34.896634 13514 tablet_bootstrap.cc:654] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:34.910151 13665 raft_consensus.cc:348] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.910962 13665 raft_consensus.cc:374] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:34.911211 13665 raft_consensus.cc:729] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:34.911847 13665 consensus_queue.cc:260] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:34.878477 12698 consensus_peers.cc:487] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5b08ed744b3420f98d241ad96bca6d0. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:34.916334 13909 raft_consensus.cc:504] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.917834 13935 raft_consensus.cc:2229] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:34.919399 13909 leader_election.cc:288] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:34.920312 13935 raft_consensus.cc:2258] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:34.923266 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f791b2df0524997ab1c53a9e5cb39ca" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:34.924073 13336 raft_consensus.cc:3037] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:34.925477 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd83bbe7a1664c278098fa66b9c6895e" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:34.926193 13338 raft_consensus.cc:3037] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:34.929594 12700 leader_election.cc:341] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:34.930969 12700 leader_election.cc:341] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:34.931453 12700 leader_election.cc:302] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:34.937718 13864 raft_consensus.cc:2726] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:34.939921 13665 ts_tablet_manager.cc:1428] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.048s user 0.006s sys 0.006s
I20260810 23:32:34.940232 13691 tablet_bootstrap.cc:492] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:34.940744 13691 ts_tablet_manager.cc:1397] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.086s user 0.007s sys 0.012s
I20260810 23:32:34.946179 13691 raft_consensus.cc:348] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:34.959554 13691 raft_consensus.cc:374] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:34.959882 13691 raft_consensus.cc:729] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
W20260810 23:32:34.964110 12700 consensus_peers.cc:487] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 -> Peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Couldn't send request to peer f26517150f1c480a877f15aec71d020b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5b08ed744b3420f98d241ad96bca6d0. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:34.965360 13864 consensus_queue.cc:1035] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:32:34.969043 12235 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dce0ec83ed8943a0b0788505bb75d1df consensus state has no leader
I20260810 23:32:34.971486 13691 consensus_queue.cc:260] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:34.976389 12234 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 23) took 809 ms (client timeout 1000 ms). Trace:
I20260810 23:32:34.979223 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b08ed744b3420f98d241ad96bca6d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d0d9227d8574de8bd73041f3e8043a0) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:34.980726 12234 rpcz_store.cc:269] 0810 23:32:34.167309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:34.237815 (+ 70506us) service_pool.cc:227] Handling call
0810 23:32:34.298705 (+ 60890us) spinlock_profiling.cc:243] Waited 28.5 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:34.976284 (+677579us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":4,"spinlock_wait_cycles":65462656}
I20260810 23:32:34.982980 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd83bbe7a1664c278098fa66b9c6895e" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
W20260810 23:32:34.985723 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet e5b08ed744b3420f98d241ad96bca6d0 because the tablet was not found. No further retry: Not found: Tablet not found: e5b08ed744b3420f98d241ad96bca6d0
I20260810 23:32:34.995277 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_3e86c4915aac433bb544495b6bd22c1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:34.997123 12240 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_d318aced7c0540b3b58280f0285ecf87" } new_table_name: "default.Table_26bf2c7d86704b5b981ecebb4e90ba75" modify_external_catalogs: true
I20260810 23:32:35.001678 13944 tablet.cc:1722] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:34.966670 13213 leader_election.cc:302] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
W20260810 23:32:35.005280 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e86c4915aac433bb544495b6bd22c1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:35.010502 13868 raft_consensus.cc:504] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.013397 13691 ts_tablet_manager.cc:1428] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.072s user 0.009s sys 0.000s
I20260810 23:32:35.014639 13691 tablet_bootstrap.cc:492] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:35.017477 13943 raft_consensus.cc:2781] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:34.991040 13866 raft_consensus.cc:504] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.022369 13665 tablet_bootstrap.cc:492] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:35.024464 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f791b2df0524997ab1c53a9e5cb39ca" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:35.031426 13935 ts_tablet_manager.cc:1905] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:35.033690 13336 raft_consensus.cc:2454] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:35.048687 12229 master_service.cc:946] ReplaceTablet: received request to replace tablet 18b29cda94584badb1225477ec2bd1bc from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:35.051299 13691 tablet_bootstrap.cc:654] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:35.059720 13514 tablet_bootstrap.cc:492] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:35.060304 13514 ts_tablet_manager.cc:1397] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.222s user 0.023s sys 0.013s
I20260810 23:32:35.062691 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1f7c16b6bc4dc0bac80ac55332f056" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:35.063477 13062 raft_consensus.cc:3037] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:35.059791 12700 leader_election.cc:341] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:35.070091 13514 raft_consensus.cc:348] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:35.071341 13514 raft_consensus.cc:374] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:35.071741 13514 raft_consensus.cc:729] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:35.071830 12698 leader_election.cc:302] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:32:35.072620 13514 consensus_queue.cc:260] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:35.073218 12698 raft_consensus.cc:2670] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260810 23:32:35.075027 13514 ts_tablet_manager.cc:1428] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.014s user 0.003s sys 0.002s
I20260810 23:32:35.090734 13514 tablet_bootstrap.cc:492] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:35.053190 13338 raft_consensus.cc:2454] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:35.104663 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1f7c16b6bc4dc0bac80ac55332f056" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:35.106163 12806 raft_consensus.cc:3037] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:35.118680 12698 leader_election.cc:302] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:32:35.120911 13909 raft_consensus.cc:2781] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:35.121799 13909 raft_consensus.cc:686] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Becoming Leader. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Running, Role: LEADER
I20260810 23:32:35.123773 13909 consensus_queue.cc:237] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.062049 13866 leader_election.cc:288] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:35.108351 13943 raft_consensus.cc:686] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:35.131142 13943 consensus_queue.cc:237] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.153071 13936 raft_consensus.cc:482] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:35.157266 13868 leader_election.cc:288] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:35.138736 13514 tablet_bootstrap.cc:654] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:35.163408 12224 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 2ebe029d24784cb8adc74ec0f383fe9d
I20260810 23:32:35.167301 12237 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 88c53785fdd24df79f138df6176e75d2 (table default.Table_bacbea393b664097a0b52af2ce5292de [id=71c9424f327040a68889fc257935c535]). Expected version 1 got 0
I20260810 23:32:35.165875 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79da9583d3524347a9be09e35affaaad" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:35.168825 12805 raft_consensus.cc:3037] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:35.169589 13665 tablet_bootstrap.cc:654] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:35.174932 12224 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_a0d143fe7e1947e6b947eda2d6042925" } new_table_name: "default.Table_d8a0e3e655b9417d9df3ff6d82b76f15" modify_external_catalogs: true
I20260810 23:32:35.178851 12805 raft_consensus.cc:2454] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:35.182337 13864 consensus_queue.cc:1035] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:35.186234 13213 leader_election.cc:302] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:35.187247 13868 raft_consensus.cc:2781] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:35.187857 13868 raft_consensus.cc:686] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:35.189823 13868 consensus_queue.cc:237] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.194387 12234 master_service.cc:946] ReplaceTablet: received request to replace tablet 3d596299c82b4ff1ac151d68b97ff55b from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:35.194804 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet 79da9583d3524347a9be09e35affaaad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:35.204885 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79da9583d3524347a9be09e35affaaad" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:35.204707 13936 raft_consensus.cc:504] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:35.207947 13955 log.cc:924] Time spent T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
W20260810 23:32:35.209442 13212 leader_election.cc:341] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:35.213382 13974 tablet_replica.cc:330] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:35.035372 13846 raft_consensus.cc:504] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.214242 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b08ed744b3420f98d241ad96bca6d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d0d9227d8574de8bd73041f3e8043a0) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:35.218384 12806 raft_consensus.cc:2454] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:35.221796 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_cdd5220b787349159091ea6dc3359c9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:35.223961 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdd5220b787349159091ea6dc3359c9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:35.225684 13936 leader_election.cc:288] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:35.203891 13062 raft_consensus.cc:2454] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:35.238427 13846 leader_election.cc:288] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:35.239555 13060 raft_consensus.cc:2379] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 191d120f4c7f44fc8475305a6d2e75f3 in current term 1: Already voted for candidate f26517150f1c480a877f15aec71d020b in this term.
W20260810 23:32:35.247088 12237 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 16) took 930 ms (client timeout 1000 ms). Trace:
I20260810 23:32:35.258896 13691 tablet_bootstrap.cc:492] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:35.259862 13691 ts_tablet_manager.cc:1397] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.245s user 0.015s sys 0.014s
I20260810 23:32:35.262650 12228 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet c892e6bc117a4fff917342b8eefaf74d
W20260810 23:32:35.269613 12237 rpcz_store.cc:269] 0810 23:32:34.316454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:34.316758 (+ 304us) service_pool.cc:227] Handling call
0810 23:32:34.369730 (+ 52972us) spinlock_profiling.cc:243] Waited 47.2 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:35.246938 (+877208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25829,"reported_tablets":6,"spinlock_wait_cycles":108671744}
I20260810 23:32:35.274127 13691 raft_consensus.cc:348] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.278010 13691 raft_consensus.cc:374] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:35.280156 13691 raft_consensus.cc:729] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
W20260810 23:32:35.280795 12226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 39) took 1165 ms (client timeout 1000 ms). Trace:
I20260810 23:32:35.286376 13691 consensus_queue.cc:260] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.288353 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_69fa732a88234f859e1cbb8d706d7c76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:35.292599 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0d6cd2ad1d4a65929b0d856a171592" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:35.294075 12805 raft_consensus.cc:2379] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:32:35.293896 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd83bbe7a1664c278098fa66b9c6895e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:35.298346 13514 tablet_bootstrap.cc:492] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:35.298898 13514 ts_tablet_manager.cc:1397] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.208s user 0.022s sys 0.025s
I20260810 23:32:35.301898 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e26fe4d42445279f12bb17b2841d08 with delete_type TABLET_DATA_DELETED (replaced by tablet b261673faa964b65af3576c89e302864) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:35.303683 12768 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:35.304248 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet 79da9583d3524347a9be09e35affaaad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:35.303426 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0d6cd2ad1d4a65929b0d856a171592 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:35.308534 13764 consensus_queue.cc:1035] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:32:35.318130 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69fa732a88234f859e1cbb8d706d7c76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:35.320864 13691 ts_tablet_manager.cc:1428] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.060s user 0.007s sys 0.003s
I20260810 23:32:35.329592 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0d6cd2ad1d4a65929b0d856a171592" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:35.337113 13514 raft_consensus.cc:348] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:35.338228 13514 raft_consensus.cc:374] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:35.338542 13514 raft_consensus.cc:729] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:35.339330 13514 consensus_queue.cc:260] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:35.352123 13691 tablet_bootstrap.cc:492] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:35.351549 12952 leader_election.cc:302] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:35.353020 13846 raft_consensus.cc:3037] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:35.356104 13514 ts_tablet_manager.cc:1428] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.057s user 0.010s sys 0.003s
I20260810 23:32:35.360589 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0d6cd2ad1d4a65929b0d856a171592 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:35.365621 13974 raft_consensus.cc:2229] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:35.366974 13974 pending_rounds.cc:62] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3: Trying to abort 1 pending ops.
I20260810 23:32:35.367233 13935 ts_tablet_manager.cc:1918] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:35.367302 13974 pending_rounds.cc:68] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317113877273448448 op_type: NO_OP noop_request { }
I20260810 23:32:35.367774 13935 log.cc:1196] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/1f791b2df0524997ab1c53a9e5cb39ca
I20260810 23:32:35.357546 13213 leader_election.cc:302] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:35.372170 13943 raft_consensus.cc:2781] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:35.372687 13943 raft_consensus.cc:686] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:35.373055 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:35.330634 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd83bbe7a1664c278098fa66b9c6895e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:35.374361 13335 raft_consensus.cc:2379] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate 191d120f4c7f44fc8475305a6d2e75f3 in this term.
I20260810 23:32:35.374068 13943 consensus_queue.cc:237] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.390875 13866 raft_consensus.cc:482] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:35.391420 13866 raft_consensus.cc:504] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.393779 13866 leader_election.cc:288] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:35.394680 13864 raft_consensus.cc:482] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:35.395227 13864 raft_consensus.cc:504] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.367884 13974 raft_consensus.cc:2866] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:32:35.396508 13974 raft_consensus.cc:2258] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:35.401621 13864 leader_election.cc:288] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:35.403558 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd110664bac34f52bfe922a3a76c6290" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:35.405951 12698 leader_election.cc:341] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:35.407447 13514 tablet_bootstrap.cc:492] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:35.410730 13974 ts_tablet_manager.cc:1905] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:35.420199 12952 leader_election.cc:302] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:35.422974 13691 tablet_bootstrap.cc:654] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:35.425514 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2f8688fc6046d58009800c5e288302" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:35.426730 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd110664bac34f52bfe922a3a76c6290" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:35.427928 13936 raft_consensus.cc:2726] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260810 23:32:35.429534 13212 leader_election.cc:341] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:35.443818 12700 leader_election.cc:341] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:35.446683 13935 ts_tablet_manager.cc:1939] T 1f791b2df0524997ab1c53a9e5cb39ca P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
W20260810 23:32:35.446511 13549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:35.447558 13549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260810 23:32:35.447906 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2f8688fc6046d58009800c5e288302" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:35.449883 12806 raft_consensus.cc:2454] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:35.444691 12700 leader_election.cc:302] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:35.454725 13864 raft_consensus.cc:2726] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:35.456765 13935 tablet_replica.cc:330] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:35.458564 13935 raft_consensus.cc:2229] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:35.461759 13213 leader_election.cc:302] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
W20260810 23:32:35.462848 13549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:32:35.465672 13866 raft_consensus.cc:2781] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:35.466218 13866 raft_consensus.cc:482] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:35.466794 13866 raft_consensus.cc:3037] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:35.473470 13665 tablet_bootstrap.cc:492] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:35.474038 13665 ts_tablet_manager.cc:1397] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.469s user 0.034s sys 0.028s
I20260810 23:32:35.486032 13846 raft_consensus.cc:2726] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:35.490383 13665 raft_consensus.cc:348] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.491175 13665 raft_consensus.cc:374] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:35.491466 13665 raft_consensus.cc:729] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:35.492340 13665 consensus_queue.cc:260] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:35.281512 12226 rpcz_store.cc:269] 0810 23:32:34.115287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:34.116001 (+ 714us) service_pool.cc:227] Handling call
0810 23:32:34.190870 (+ 74869us) spinlock_profiling.cc:243] Waited 46 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:35.280651 (+1089781us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:34.997875 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:34.999797 (+ 1922us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:34.999832 (+ 35us) write_op.cc:435] Acquired schema lock
0810 23:32:34.999843 (+ 11us) tablet.cc:660] Decoding operations
0810 23:32:35.000575 (+ 732us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:35.000655 (+ 80us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:35.000709 (+ 54us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:35.001218 (+ 509us) tablet.cc:699] Row locks acquired
0810 23:32:35.001248 (+ 30us) write_op.cc:260] PREPARE: finished
0810 23:32:35.001520 (+ 272us) write_op.cc:270] Start()
0810 23:32:35.001739 (+ 219us) write_op.cc:276] Timestamp: P: 1786404755001430 usec, L: 0
0810 23:32:35.001756 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:32:35.002916 (+ 1160us) log.cc:842] Serialized 919 byte log entry
0810 23:32:35.156664 (+153748us) op_driver.cc:444] REPLICATION: finished
0810 23:32:35.157110 (+ 446us) write_op.cc:301] APPLY: starting
0810 23:32:35.157190 (+ 80us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:35.157391 (+ 201us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:35.157552 (+ 161us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:35.158039 (+ 487us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:35.158278 (+ 239us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:35.158356 (+ 78us) write_op.cc:312] APPLY: finished
0810 23:32:35.158683 (+ 327us) log.cc:842] Serialized 29 byte log entry
0810 23:32:35.158871 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:35.158963 (+ 92us) write_op.cc:454] Released schema lock
0810 23:32:35.159010 (+ 47us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:35.159258 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:35.159291 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":25476,"reported_tablets":10,"spinlock_wait_cycles":105757056,"child_traces":[["op",{"apply.queue_time_us":143,"apply.run_cpu_time_us":2460,"apply.run_wall_time_us":2460,"num_ops":2,"prepare.queue_time_us":566663,"prepare.run_cpu_time_us":4828,"prepare.run_wall_time_us":82160,"raft.queue_time_us":194,"raft.run_cpu_time_us":855,"raft.run_wall_time_us":856,"replication_time_us":154637}]]}
I20260810 23:32:35.492798 13691 tablet_bootstrap.cc:492] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:35.499135 13691 ts_tablet_manager.cc:1397] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.147s user 0.015s sys 0.005s
I20260810 23:32:35.506846 13935 raft_consensus.cc:2258] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:35.512075 13665 ts_tablet_manager.cc:1428] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.038s user 0.002s sys 0.011s
I20260810 23:32:35.513383 13665 tablet_bootstrap.cc:492] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:35.526244 13514 tablet_bootstrap.cc:654] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:35.539475 13935 ts_tablet_manager.cc:1905] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:35.548978 13866 raft_consensus.cc:504] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:35.538684 12240 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d697e7533af4fc0a15cb41f3fcce4db has no consensus state
I20260810 23:32:35.571316 13691 raft_consensus.cc:348] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.572926 13691 raft_consensus.cc:374] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:35.573452 13691 raft_consensus.cc:729] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:35.580446 12240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e0603d4692934dce9e07597811ccad0a" } modify_external_catalogs: true
I20260810 23:32:35.581272 12240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e0603d4692934dce9e07597811ccad0a" } modify_external_catalogs: true
I20260810 23:32:35.598979 13691 consensus_queue.cc:260] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.602304 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2f8688fc6046d58009800c5e288302" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:35.603219 13866 leader_election.cc:288] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:35.603350 12806 raft_consensus.cc:3037] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:35.579021 13665 tablet_bootstrap.cc:654] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:35.608747 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2f8688fc6046d58009800c5e288302" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
W20260810 23:32:35.614102 13212 leader_election.cc:341] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:35.615331 13944 tablet.cc:1722] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:32:35.621567 13691 ts_tablet_manager.cc:1428] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.122s user 0.008s sys 0.000s
I20260810 23:32:35.637554 13691 tablet_bootstrap.cc:492] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:35.638109 13985 tablet.cc:1722] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:32:35.641772 13866 raft_consensus.cc:482] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:35.642978 13866 raft_consensus.cc:504] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.650182 13514 tablet_bootstrap.cc:492] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:35.650769 13514 ts_tablet_manager.cc:1397] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.291s user 0.019s sys 0.007s
I20260810 23:32:35.655086 13514 raft_consensus.cc:348] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:35.655814 13514 raft_consensus.cc:374] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:35.656097 13514 raft_consensus.cc:729] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:35.661571 12235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_adf76678e17a494185985a544def0f2d" } modify_external_catalogs: true
I20260810 23:32:35.662778 12235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_adf76678e17a494185985a544def0f2d" } modify_external_catalogs: true
I20260810 23:32:35.673365 13764 raft_consensus.cc:482] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:35.673990 13764 raft_consensus.cc:504] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:35.677747 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d596299c82b4ff1ac151d68b97ff55b" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:35.679313 12700 leader_election.cc:341] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:35.680634 13764 leader_election.cc:288] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:35.690789 13866 leader_election.cc:288] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:35.692062 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c45b71432a45269cd6a106fc5f7292" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:35.692899 13059 raft_consensus.cc:2454] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:35.694631 13212 leader_election.cc:302] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:35.677517 13514 consensus_queue.cc:260] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:35.696069 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d596299c82b4ff1ac151d68b97ff55b" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:35.704562 13866 raft_consensus.cc:2781] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:35.705307 13866 raft_consensus.cc:482] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:35.705989 13866 raft_consensus.cc:3037] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:35.714792 13665 tablet_bootstrap.cc:492] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:35.715294 13665 ts_tablet_manager.cc:1397] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.202s user 0.012s sys 0.008s
W20260810 23:32:35.698582 12698 leader_election.cc:341] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:35.716364 12698 leader_election.cc:302] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:35.717033 13514 ts_tablet_manager.cc:1428] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.066s user 0.007s sys 0.004s
I20260810 23:32:35.718703 13514 tablet_bootstrap.cc:492] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:35.720002 13909 raft_consensus.cc:2726] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:35.720839 12224 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab8d987109ba40198645881b449cb972 has no consensus state
I20260810 23:32:35.693871 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c45b71432a45269cd6a106fc5f7292" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:35.724263 12805 raft_consensus.cc:2454] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
W20260810 23:32:35.724516 12214 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 3d596299c82b4ff1ac151d68b97ff55b (table default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3d596299c82b4ff1ac151d68b97ff55b consensus state has no leader
I20260810 23:32:35.725570 13943 raft_consensus.cc:482] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:35.726460 13909 raft_consensus.cc:482] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:35.726254 13943 raft_consensus.cc:504] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.727190 13909 raft_consensus.cc:504] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:35.730154 13909 leader_election.cc:288] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:35.733980 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8986e59b2314bd69b9b1de36f1f2ba7" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:35.735544 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8986e59b2314bd69b9b1de36f1f2ba7" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:35.737711 12700 leader_election.cc:341] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:35.738490 13943 leader_election.cc:288] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:35.739452 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd110664bac34f52bfe922a3a76c6290" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:35.741448 13212 leader_election.cc:341] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:35.741654 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_cd333cc49a4b49178463f3c996caac95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:35.743703 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd333cc49a4b49178463f3c996caac95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:35.744465 12698 leader_election.cc:341] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:35.745359 12698 leader_election.cc:302] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:35.747066 13764 raft_consensus.cc:2726] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:35.750422 13665 raft_consensus.cc:348] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.751392 13665 raft_consensus.cc:374] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:35.751724 13665 raft_consensus.cc:729] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:35.752573 13665 consensus_queue.cc:260] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.754639 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd110664bac34f52bfe922a3a76c6290" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:35.755610 12805 raft_consensus.cc:2454] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:35.747478 13691 tablet_bootstrap.cc:654] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:35.757777 13213 leader_election.cc:302] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:32:35.769263 13974 ts_tablet_manager.cc:1918] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:35.776515 13868 raft_consensus.cc:2781] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:35.776989 13868 raft_consensus.cc:482] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:35.777388 13868 raft_consensus.cc:3037] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:35.778471 13665 ts_tablet_manager.cc:1428] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.063s user 0.004s sys 0.004s
I20260810 23:32:35.779625 13665 tablet_bootstrap.cc:492] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:35.784175 13974 log.cc:1196] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/79da9583d3524347a9be09e35affaaad
I20260810 23:32:35.790108 13974 ts_tablet_manager.cc:1939] T 79da9583d3524347a9be09e35affaaad P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:35.802569 13868 raft_consensus.cc:504] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.808647 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd110664bac34f52bfe922a3a76c6290" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:35.809754 13868 leader_election.cc:288] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:35.811451 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd110664bac34f52bfe922a3a76c6290" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
W20260810 23:32:35.813849 13212 leader_election.cc:341] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:35.814245 13691 tablet_bootstrap.cc:492] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:35.814682 13691 ts_tablet_manager.cc:1397] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.177s user 0.009s sys 0.011s
I20260810 23:32:35.814798 12805 raft_consensus.cc:3037] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:35.820101 13514 tablet_bootstrap.cc:654] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:35.826362 13935 ts_tablet_manager.cc:1918] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:35.826490 12224 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_969f8ae489e24f139b305369d629d7af" } new_table_name: "default.Table_b57f0f6e94384d22af5d3bf7817d5044" modify_external_catalogs: true
I20260810 23:32:35.833961 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 79da9583d3524347a9be09e35affaaad (table default.Table_275ea19d22624a12a6b3b2c8e985c4af [id=5b4476e15a7e410599b70f84ab267de9]) successfully deleted
W20260810 23:32:35.836022 12215 consensus_peers.cc:487] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 -> Peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Couldn't send request to peer 198f534261994047a68e1d509bb2fd8b. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260810 23:32:35.839746 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet e5b08ed744b3420f98d241ad96bca6d0 because the tablet was not found. No further retry: Not found: Tablet not found: e5b08ed744b3420f98d241ad96bca6d0
I20260810 23:32:35.843714 13691 raft_consensus.cc:348] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.844550 13691 raft_consensus.cc:374] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:35.852387 13935 log.cc:1196] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/e5b08ed744b3420f98d241ad96bca6d0
I20260810 23:32:35.844853 13691 raft_consensus.cc:729] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:35.857273 13935 ts_tablet_manager.cc:1939] T e5b08ed744b3420f98d241ad96bca6d0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:35.857136 13691 consensus_queue.cc:260] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.843216 13665 tablet_bootstrap.cc:654] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:35.843863 13974 tablet_replica.cc:330] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:35.863224 13974 raft_consensus.cc:2229] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:35.864714 13974 pending_rounds.cc:62] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3: Trying to abort 1 pending ops.
I20260810 23:32:35.866314 13974 pending_rounds.cc:68] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317113877044219904 op_type: NO_OP noop_request { }
I20260810 23:32:35.856127 13989 tablet.cc:1722] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:32:35.871614 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 66) took 1155 ms. Trace:
I20260810 23:32:35.831910 13866 raft_consensus.cc:504] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.876819 13691 ts_tablet_manager.cc:1428] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.062s user 0.006s sys 0.003s
I20260810 23:32:35.877076 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c45b71432a45269cd6a106fc5f7292" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:35.877880 13059 raft_consensus.cc:3037] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:35.878650 13691 tablet_bootstrap.cc:492] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:35.885879 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_2c509bc8585b4ca68502ce3897fa18fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:35.889401 13059 raft_consensus.cc:2454] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
W20260810 23:32:35.890838 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c509bc8585b4ca68502ce3897fa18fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:35.891134 13212 leader_election.cc:302] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:35.892175 13868 raft_consensus.cc:2781] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:35.892818 13868 raft_consensus.cc:686] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:35.894207 13868 consensus_queue.cc:237] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.921391 13846 raft_consensus.cc:482] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:35.922242 13846 raft_consensus.cc:504] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.923621 13974 raft_consensus.cc:2866] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:32:35.924417 13974 raft_consensus.cc:2258] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:35.927842 13974 ts_tablet_manager.cc:1905] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:35.932348 13846 leader_election.cc:288] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:35.932229 13866 leader_election.cc:288] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:32:35.937849 12952 leader_election.cc:341] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 79da9583d3524347a9be09e35affaaad
I20260810 23:32:35.876099 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c45b71432a45269cd6a106fc5f7292" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:35.951107 12803 raft_consensus.cc:3037] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:35.951558 13280 tablet_service.cc:1467] Processing CreateTablet for tablet 13ad5d63a2104995990a86a000963cdc (DEFAULT_TABLE table=default.Table_34eb8fd429274b5d8af145c70aab107b [id=dae244793fd24816a8199c38f23a25b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:35.953032 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ad5d63a2104995990a86a000963cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:35.847082 12805 raft_consensus.cc:2454] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:35.955200 13213 leader_election.cc:302] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:32:35.955902 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 13ad5d63a2104995990a86a000963cdc (DEFAULT_TABLE table=default.Table_34eb8fd429274b5d8af145c70aab107b [id=dae244793fd24816a8199c38f23a25b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:35.957298 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ad5d63a2104995990a86a000963cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:35.958544 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 42517f11422b42c39db282e2c5f584a3 (DEFAULT_TABLE table=default.Table_36d49207cc33452db87dd140a5193dc6 [id=326b0d80eb584c2b904f2167629dc30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:35.959757 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42517f11422b42c39db282e2c5f584a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:35.960130 13935 rpcz_store.cc:276] 0810 23:32:34.716695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:34.756395 (+ 39700us) service_pool.cc:227] Handling call
0810 23:32:35.456363 (+699968us) ts_tablet_manager.cc:1145] Deleting tablet e5b08ed744b3420f98d241ad96bca6d0
0810 23:32:35.456460 (+ 97us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:35.648981 (+192521us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:35.825946 (+176965us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:35.871524 (+ 45578us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":679401}
I20260810 23:32:35.961010 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 13ad5d63a2104995990a86a000963cdc (DEFAULT_TABLE table=default.Table_34eb8fd429274b5d8af145c70aab107b [id=dae244793fd24816a8199c38f23a25b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:35.962260 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ad5d63a2104995990a86a000963cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:35.964707 13866 raft_consensus.cc:2781] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:35.964826 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet e5b08ed744b3420f98d241ad96bca6d0 (table default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]) successfully deleted
I20260810 23:32:35.965442 13866 raft_consensus.cc:686] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:35.969117 13023 tablet_service.cc:1467] Processing CreateTablet for tablet 42517f11422b42c39db282e2c5f584a3 (DEFAULT_TABLE table=default.Table_36d49207cc33452db87dd140a5193dc6 [id=326b0d80eb584c2b904f2167629dc30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:35.971046 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42517f11422b42c39db282e2c5f584a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:35.929762 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79da9583d3524347a9be09e35affaaad" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:35.974289 12802 raft_consensus.cc:2379] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:32:35.976208 12958 leader_election.cc:302] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:35.983518 13514 tablet_bootstrap.cc:492] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:35.984230 13514 ts_tablet_manager.cc:1397] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.266s user 0.018s sys 0.002s
I20260810 23:32:35.984313 13846 raft_consensus.cc:3037] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:35.962882 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 67) took 1194 ms. Trace:
I20260810 23:32:35.988257 13935 rpcz_store.cc:276] 0810 23:32:34.768246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:34.777675 (+ 9429us) service_pool.cc:227] Handling call
0810 23:32:35.962168 (+1184493us) ts_tablet_manager.cc:1145] Deleting tablet 081b4e1f53c74f768a4bf401ae22b585
0810 23:32:35.962218 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:35.962789 (+ 571us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1183808}
I20260810 23:32:35.966795 13866 consensus_queue.cc:237] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:35.992653 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 42517f11422b42c39db282e2c5f584a3 (DEFAULT_TABLE table=default.Table_36d49207cc33452db87dd140a5193dc6 [id=326b0d80eb584c2b904f2167629dc30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:35.994065 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42517f11422b42c39db282e2c5f584a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:35.967183 12803 raft_consensus.cc:2454] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
W20260810 23:32:35.999032 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 081b4e1f53c74f768a4bf401ae22b585 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 081b4e1f53c74f768a4bf401ae22b585 already in progress: creating tablet
I20260810 23:32:35.999598 13935 tablet_replica.cc:330] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:36.003098 12229 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e8c26e9362704b6fb84bba331ef685ea" } new_table_name: "default.Table_72495aefc75943eeb8a43dd95ba2b03d" modify_external_catalogs: true
I20260810 23:32:36.003882 13665 tablet_bootstrap.cc:492] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:36.004422 13665 ts_tablet_manager.cc:1397] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.225s user 0.012s sys 0.003s
I20260810 23:32:36.008173 13514 raft_consensus.cc:348] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:36.009109 13514 raft_consensus.cc:374] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:36.009419 13514 raft_consensus.cc:729] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:36.010193 13514 consensus_queue.cc:260] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:36.012318 13514 ts_tablet_manager.cc:1428] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.028s user 0.007s sys 0.002s
I20260810 23:32:36.013290 13514 tablet_bootstrap.cc:492] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:36.017380 13935 raft_consensus.cc:2229] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:36.023824 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 6a3a834d77474c3d8feaa5a01f3faff3 (DEFAULT_TABLE table=default.Table_0d38f0b73fd746b4a487e7b1600eae77 [id=1ea3dfab929845f89d4680129ae45d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.025748 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a3a834d77474c3d8feaa5a01f3faff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.034936 13864 consensus_queue.cc:1035] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:36.043871 13935 raft_consensus.cc:2258] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:36.045437 13665 raft_consensus.cc:348] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:36.036752 13514 tablet_bootstrap.cc:654] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:36.048379 13665 raft_consensus.cc:374] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:36.053043 13665 raft_consensus.cc:729] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
W20260810 23:32:36.055140 12217 catalog_manager.cc:4914] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): alter failed for tablet e5b08ed744b3420f98d241ad96bca6d0 (table default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]),no further retry: Not found: Tablet not found: e5b08ed744b3420f98d241ad96bca6d0
I20260810 23:32:35.942454 12806 raft_consensus.cc:2454] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:36.061484 13864 consensus_queue.cc:1035] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:36.063378 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 133654008f5247b2ac87dcd09195f197 (DEFAULT_TABLE table=default.Table_d664d07d116e4f2aa4fc263449eb8dd0 [id=04ed6428e4df49b2ab0438035da70731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.065521 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133654008f5247b2ac87dcd09195f197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.065662 13213 leader_election.cc:302] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:32:36.067462 13868 raft_consensus.cc:2781] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:36.069741 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 6a3a834d77474c3d8feaa5a01f3faff3 (DEFAULT_TABLE table=default.Table_0d38f0b73fd746b4a487e7b1600eae77 [id=1ea3dfab929845f89d4680129ae45d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.053804 13665 consensus_queue.cc:260] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:36.055096 13935 ts_tablet_manager.cc:1905] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:36.078655 12761 tablet_service.cc:1467] Processing CreateTablet for tablet 6a3a834d77474c3d8feaa5a01f3faff3 (DEFAULT_TABLE table=default.Table_0d38f0b73fd746b4a487e7b1600eae77 [id=1ea3dfab929845f89d4680129ae45d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.080488 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a3a834d77474c3d8feaa5a01f3faff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.071797 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a3a834d77474c3d8feaa5a01f3faff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.088950 13868 raft_consensus.cc:686] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:36.090695 13868 consensus_queue.cc:237] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:36.122965 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 133654008f5247b2ac87dcd09195f197 (DEFAULT_TABLE table=default.Table_d664d07d116e4f2aa4fc263449eb8dd0 [id=04ed6428e4df49b2ab0438035da70731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.123687 13514 tablet_bootstrap.cc:492] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:36.124316 13514 ts_tablet_manager.cc:1397] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.111s user 0.011s sys 0.012s
I20260810 23:32:36.125306 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133654008f5247b2ac87dcd09195f197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.125111 13846 raft_consensus.cc:2726] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:36.144583 13514 raft_consensus.cc:348] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:36.145465 13514 raft_consensus.cc:374] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:36.145753 13514 raft_consensus.cc:729] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:36.146469 13514 consensus_queue.cc:260] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:36.158075 12224 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 874ca0c3586b41a8a3074d8988813633 consensus state has no leader
I20260810 23:32:36.169483 13864 raft_consensus.cc:482] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:36.170212 13864 raft_consensus.cc:504] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:36.172636 13514 ts_tablet_manager.cc:1428] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.048s user 0.009s sys 0.000s
I20260810 23:32:36.174402 13514 tablet_bootstrap.cc:492] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:36.180365 13691 tablet_bootstrap.cc:654] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:36.203689 13025 tablet_service.cc:1467] Processing CreateTablet for tablet db363eb0f98d460eb6f5a577233d8ac0 (DEFAULT_TABLE table=default.Table_e81793ce40c14cfe81a319f4cbd088bf [id=cca554c88a7844d5aa3312b12e89369e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.205727 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db363eb0f98d460eb6f5a577233d8ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.210762 13665 ts_tablet_manager.cc:1428] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.206s user 0.008s sys 0.010s
I20260810 23:32:36.216768 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 133654008f5247b2ac87dcd09195f197 (DEFAULT_TABLE table=default.Table_d664d07d116e4f2aa4fc263449eb8dd0 [id=04ed6428e4df49b2ab0438035da70731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.218276 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133654008f5247b2ac87dcd09195f197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.237329 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b29cda94584badb1225477ec2bd1bc" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:36.239547 12700 leader_election.cc:341] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:36.242120 13864 leader_election.cc:288] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:36.244591 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b29cda94584badb1225477ec2bd1bc" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:36.247197 12698 leader_election.cc:341] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:36.248070 13514 tablet_bootstrap.cc:654] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:36.247965 12698 leader_election.cc:302] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:36.249696 13864 raft_consensus.cc:2726] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:36.268286 13514 tablet_bootstrap.cc:492] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:36.268935 13514 ts_tablet_manager.cc:1397] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.095s user 0.010s sys 0.008s
I20260810 23:32:36.273429 13514 raft_consensus.cc:348] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:36.273495 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_83e4e8027f8b4f61b104e94ccf146481"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:36.274211 13514 raft_consensus.cc:374] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:36.274448 13514 raft_consensus.cc:729] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:36.275090 13514 consensus_queue.cc:260] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:36.280953 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83e4e8027f8b4f61b104e94ccf146481 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:36.286106 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_7548a45d866747b9b0944e13b211c643"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:36.289330 13514 ts_tablet_manager.cc:1428] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
W20260810 23:32:36.289849 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7548a45d866747b9b0944e13b211c643 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:36.303701 13514 tablet_bootstrap.cc:492] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:36.304874 13868 consensus_queue.cc:1035] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:36.307590 12761 tablet_service.cc:1467] Processing CreateTablet for tablet db363eb0f98d460eb6f5a577233d8ac0 (DEFAULT_TABLE table=default.Table_e81793ce40c14cfe81a319f4cbd088bf [id=cca554c88a7844d5aa3312b12e89369e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.309581 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db363eb0f98d460eb6f5a577233d8ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.319458 12772 tablet_service.cc:1467] Processing CreateTablet for tablet 440de7d9de8d4be9a5322e1fe3388b9b (DEFAULT_TABLE table=default.Table_529a79b85b6f4c5da1ab44678a564943 [id=c56022cc8cad4c7cbefafad0782cefde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.321359 12772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440de7d9de8d4be9a5322e1fe3388b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.331106 13691 tablet_bootstrap.cc:492] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:36.331894 13691 ts_tablet_manager.cc:1397] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.453s user 0.011s sys 0.005s
I20260810 23:32:36.350849 12240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 840400119d20424ea0f03f2bff550d44
I20260810 23:32:36.354326 12766 tablet_service.cc:1514] Processing DeleteTablet for tablet 840400119d20424ea0f03f2bff550d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:36.361084 13665 tablet_bootstrap.cc:492] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:36.386143 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_7ba7fd44e98542e78c604739a5086a91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:36.390578 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ba7fd44e98542e78c604739a5086a91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:36.401119 12240 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_cd333cc49a4b49178463f3c996caac95" } new_table_name: "default.Table_1e5f07908d444cf9af2cae2ea83ae7ae" modify_external_catalogs: true
I20260810 23:32:36.345342 13026 tablet_service.cc:1467] Processing CreateTablet for tablet 440de7d9de8d4be9a5322e1fe3388b9b (DEFAULT_TABLE table=default.Table_529a79b85b6f4c5da1ab44678a564943 [id=c56022cc8cad4c7cbefafad0782cefde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.412881 13026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440de7d9de8d4be9a5322e1fe3388b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.417979 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet 840400119d20424ea0f03f2bff550d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:36.420548 13868 raft_consensus.cc:482] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:36.420857 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 4d697e7533af4fc0a15cb41f3fcce4db (DEFAULT_TABLE table=default.Table_26bf2c7d86704b5b981ecebb4e90ba75 [id=cefe7b97c1db42ef8712bf3c7a359af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.421231 13021 tablet_service.cc:1467] Processing CreateTablet for tablet 423e8d583d704fa9834026ebcd57e17e (DEFAULT_TABLE table=default.Table_514d038dba3a4fc399b97e7edc254ad9 [id=53909ddaaede4deb97cec7ad02020c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.421548 13868 raft_consensus.cc:504] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:36.422688 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d697e7533af4fc0a15cb41f3fcce4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.424822 13021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 423e8d583d704fa9834026ebcd57e17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.424393 13691 raft_consensus.cc:348] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:36.425485 13691 raft_consensus.cc:374] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:36.425829 13691 raft_consensus.cc:729] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:36.436595 13868 leader_election.cc:288] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:36.429239 13691 consensus_queue.cc:260] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:36.439195 13691 ts_tablet_manager.cc:1428] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.107s user 0.003s sys 0.003s
I20260810 23:32:36.445075 13014 tablet_service.cc:1467] Processing CreateTablet for tablet 4b71d565669f40d8a6d06ab267069430 (DEFAULT_TABLE table=default.Table_93dd9072dddd48ec859e82f2c6e0244c [id=24e3b49b5b8f4bcfabc485ddf55ee01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.451963 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e26fe4d42445279f12bb17b2841d08" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:36.455623 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 4d697e7533af4fc0a15cb41f3fcce4db (DEFAULT_TABLE table=default.Table_26bf2c7d86704b5b981ecebb4e90ba75 [id=cefe7b97c1db42ef8712bf3c7a359af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.457091 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d697e7533af4fc0a15cb41f3fcce4db. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:36.466923 13213 leader_election.cc:341] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: Tablet not RUNNING: STOPPED
I20260810 23:32:36.469390 12234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 3d596299c82b4ff1ac151d68b97ff55b
I20260810 23:32:36.471105 12234 catalog_manager.cc:6565] ReplaceTablet: tablet 3d596299c82b4ff1ac151d68b97ff55b deleted and replaced by tablet 4cdec13ee70d402fa3a08119e7bf1313
I20260810 23:32:36.471725 12234 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 148) took 1278 ms. Trace:
I20260810 23:32:36.483098 13017 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d596299c82b4ff1ac151d68b97ff55b with delete_type TABLET_DATA_DELETED (replaced by tablet 4cdec13ee70d402fa3a08119e7bf1313) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:36.486469 13017 tablet_service.cc:1467] Processing CreateTablet for tablet fd4f55ef709d473f89a16d32160eb61b (DEFAULT_TABLE table=default.Table_addbc3e258944d0cb5d2e7fb11214412 [id=d4e32fc5299547268eb0c93e2d3c00e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.488979 12223 catalog_manager.cc:5495] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129). New cstate: current_term: 1 leader_uuid: "191d120f4c7f44fc8475305a6d2e75f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } }
W20260810 23:32:36.458943 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 75214.3us
I20260810 23:32:36.505738 13012 tablet_service.cc:1467] Processing CreateTablet for tablet ab8d987109ba40198645881b449cb972 (DEFAULT_TABLE table=default.Table_d8a0e3e655b9417d9df3ff6d82b76f15 [id=d0b86eafe9544cd98d8fce8ece3b5c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.511178 12766 tablet_service.cc:1467] Processing CreateTablet for tablet 423e8d583d704fa9834026ebcd57e17e (DEFAULT_TABLE table=default.Table_514d038dba3a4fc399b97e7edc254ad9 [id=53909ddaaede4deb97cec7ad02020c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.513872 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 423e8d583d704fa9834026ebcd57e17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.516114 12223 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:36.521366 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 4b71d565669f40d8a6d06ab267069430 (DEFAULT_TABLE table=default.Table_93dd9072dddd48ec859e82f2c6e0244c [id=24e3b49b5b8f4bcfabc485ddf55ee01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.523327 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d596299c82b4ff1ac151d68b97ff55b with delete_type TABLET_DATA_DELETED (replaced by tablet 4cdec13ee70d402fa3a08119e7bf1313) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:36.524639 12767 tablet_service.cc:1467] Processing CreateTablet for tablet fd4f55ef709d473f89a16d32160eb61b (DEFAULT_TABLE table=default.Table_addbc3e258944d0cb5d2e7fb11214412 [id=d4e32fc5299547268eb0c93e2d3c00e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.527333 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b71d565669f40d8a6d06ab267069430. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:36.533245 13214 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.11.226.190:43533, user_credentials={real_user=slave}} blocked reactor thread for 52819.3us
I20260810 23:32:36.538178 13514 tablet_bootstrap.cc:654] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:36.541409 13283 tablet_service.cc:1467] Processing CreateTablet for tablet db363eb0f98d460eb6f5a577233d8ac0 (DEFAULT_TABLE table=default.Table_e81793ce40c14cfe81a319f4cbd088bf [id=cca554c88a7844d5aa3312b12e89369e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.543465 12767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4f55ef709d473f89a16d32160eb61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.546440 13012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab8d987109ba40198645881b449cb972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.548794 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b71d565669f40d8a6d06ab267069430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.553628 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db363eb0f98d460eb6f5a577233d8ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.558600 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4f55ef709d473f89a16d32160eb61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.567129 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 440de7d9de8d4be9a5322e1fe3388b9b (DEFAULT_TABLE table=default.Table_529a79b85b6f4c5da1ab44678a564943 [id=c56022cc8cad4c7cbefafad0782cefde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.569952 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440de7d9de8d4be9a5322e1fe3388b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.572541 13010 tablet_service.cc:1467] Processing CreateTablet for tablet 9c29b11192604ac18c21b7bdc0238fb3 (DEFAULT_TABLE table=default.Table_dd2cd30e5e2349de9180be01866dc513 [id=8754da7be72b4eef80daacdcdfde82a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.574872 13010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c29b11192604ac18c21b7bdc0238fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.577131 14030 raft_consensus.cc:482] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:36.577807 13691 tablet_bootstrap.cc:492] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:36.577872 14030 raft_consensus.cc:504] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:36.583968 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet 840400119d20424ea0f03f2bff550d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:36.591813 12764 tablet_service.cc:1467] Processing CreateTablet for tablet ab8d987109ba40198645881b449cb972 (DEFAULT_TABLE table=default.Table_d8a0e3e655b9417d9df3ff6d82b76f15 [id=d0b86eafe9544cd98d8fce8ece3b5c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.594444 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab8d987109ba40198645881b449cb972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.597250 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45f9dfd03764d7fb1b6fb5e28b66bd3" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:36.478286 12234 rpcz_store.cc:276] 0810 23:32:35.193672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.194185 (+ 513us) service_pool.cc:227] Handling call
0810 23:32:35.221020 (+ 26835us) spinlock_profiling.cc:243] Waited 25.2 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e9b81 00007f0ce6e44d37 00007f0ce6e44cc2 00007f0ce6e44bf1 00007f0ce6e44b6c 00007f0ce6e43e22 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:36.471638 (+1250618us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:36.187342 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:36.188323 (+ 981us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:36.188363 (+ 40us) write_op.cc:435] Acquired schema lock
0810 23:32:36.188381 (+ 18us) tablet.cc:660] Decoding operations
0810 23:32:36.189188 (+ 807us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:36.189301 (+ 113us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:36.189351 (+ 50us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:36.189928 (+ 577us) tablet.cc:699] Row locks acquired
0810 23:32:36.189950 (+ 22us) write_op.cc:260] PREPARE: finished
0810 23:32:36.190239 (+ 289us) write_op.cc:270] Start()
0810 23:32:36.190433 (+ 194us) write_op.cc:276] Timestamp: P: 1786404756190117 usec, L: 0
0810 23:32:36.190448 (+ 15us) op_driver.cc:331] REPLICATION: starting
0810 23:32:36.342652 (+152204us) spinlock_profiling.cc:243] Waited 37 ms on lock 0x7b5400032120. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce72acdc0 00007f0ce731dbd2 00007f0ce73132d7 00007f0ce7313114 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1
0810 23:32:36.343499 (+ 847us) log.cc:842] Serialized 631 byte log entry
0810 23:32:36.375517 (+ 32018us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:36.423654 (+ 48137us) spinlock_profiling.cc:243] Waited 45.2 ms on lock 0x7b2000001078. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce72f71f5 00007f0ce73131b1 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784
0810 23:32:36.464726 (+ 41072us) op_driver.cc:444] REPLICATION: finished
0810 23:32:36.465967 (+ 1241us) write_op.cc:301] APPLY: starting
0810 23:32:36.466053 (+ 86us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:36.466216 (+ 163us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:36.466247 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:36.466999 (+ 752us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:36.467328 (+ 329us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:36.467385 (+ 57us) write_op.cc:312] APPLY: finished
0810 23:32:36.467613 (+ 228us) log.cc:842] Serialized 29 byte log entry
0810 23:32:36.467764 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:36.467862 (+ 98us) write_op.cc:454] Released schema lock
0810 23:32:36.467908 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:36.468214 (+ 306us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:36.468251 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":58042112,"child_traces":[["op",{"apply.queue_time_us":1010,"apply.run_cpu_time_us":2586,"apply.run_wall_time_us":2584,"num_ops":2,"prepare.queue_time_us":239917,"prepare.run_cpu_time_us":11661,"prepare.run_wall_time_us":236529,"raft.queue_time_us":3339,"raft.run_cpu_time_us":1411,"raft.run_wall_time_us":1861,"replication_time_us":274146,"spinlock_wait_cycles":441403776}]]}
I20260810 23:32:36.612695 12223 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet dce0ec83ed8943a0b0788505bb75d1df (table default.Table_94e24b4b27d047a99df427c453fe1752 [id=e462d54230344223a457f708501d2d30]). Expected version 3 got 0
W20260810 23:32:36.613883 12700 leader_election.cc:341] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:36.613978 12223 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 8d2f8688fc6046d58009800c5e288302 (table default.Table_783bf3bd81bb4ec294327f9ed5411afb [id=1f49f73dd9d34e968defd3a8ade21d22]). Expected version 1 got 0
I20260810 23:32:36.615160 13764 raft_consensus.cc:482] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:36.615821 13764 raft_consensus.cc:504] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:36.619830 13764 leader_election.cc:288] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:36.623451 14030 leader_election.cc:288] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:36.633980 12235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet b261673faa964b65af3576c89e302864
I20260810 23:32:36.635962 12235 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 152) took 1155 ms. Trace:
I20260810 23:32:36.641868 12763 tablet_service.cc:1514] Processing DeleteTablet for tablet b261673faa964b65af3576c89e302864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:36.637827 12235 rpcz_store.cc:276] 0810 23:32:35.481031 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.660883 (+179852us) service_pool.cc:227] Handling call
0810 23:32:35.663931 (+ 3048us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:35.664912 (+ 981us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:35.665224 (+ 312us) catalog_manager.cc:2761] Locking tablets
0810 23:32:35.929146 (+263922us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:36.593043 (+663897us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:36.593376 (+ 333us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:36.633797 (+ 40421us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:36.635886 (+ 2089us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:36.436202 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:36.437202 (+ 1000us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:36.437253 (+ 51us) write_op.cc:435] Acquired schema lock
0810 23:32:36.437297 (+ 44us) tablet.cc:660] Decoding operations
0810 23:32:36.438046 (+ 749us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:36.438091 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:36.438121 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:36.438739 (+ 618us) tablet.cc:699] Row locks acquired
0810 23:32:36.438753 (+ 14us) write_op.cc:260] PREPARE: finished
0810 23:32:36.439002 (+ 249us) write_op.cc:270] Start()
0810 23:32:36.439181 (+ 179us) write_op.cc:276] Timestamp: P: 1786404756438914 usec, L: 0
0810 23:32:36.439223 (+ 42us) op_driver.cc:331] REPLICATION: starting
0810 23:32:36.462780 (+ 23557us) log.cc:842] Serialized 765 byte log entry
0810 23:32:36.589882 (+127102us) op_driver.cc:444] REPLICATION: finished
0810 23:32:36.590521 (+ 639us) write_op.cc:301] APPLY: starting
0810 23:32:36.590647 (+ 126us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:36.590842 (+ 195us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:36.591031 (+ 189us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:36.591501 (+ 470us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:36.591780 (+ 279us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:36.591843 (+ 63us) write_op.cc:312] APPLY: finished
0810 23:32:36.592235 (+ 392us) log.cc:842] Serialized 29 byte log entry
0810 23:32:36.592500 (+ 265us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:36.592666 (+ 166us) write_op.cc:454] Released schema lock
0810 23:32:36.592709 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:36.593605 (+ 896us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:36.593643 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":237,"apply.run_cpu_time_us":2818,"apply.run_wall_time_us":3383,"num_ops":2,"prepare.queue_time_us":446748,"prepare.run_cpu_time_us":4550,"prepare.run_wall_time_us":40617,"replication_time_us":150477,"spinlock_wait_cycles":3840}]]}
I20260810 23:32:36.644639 12235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e81793ce40c14cfe81a319f4cbd088bf" } modify_external_catalogs: true
I20260810 23:32:36.646282 12235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e81793ce40c14cfe81a319f4cbd088bf" } modify_external_catalogs: true
I20260810 23:32:36.650759 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 9c29b11192604ac18c21b7bdc0238fb3 (DEFAULT_TABLE table=default.Table_dd2cd30e5e2349de9180be01866dc513 [id=8754da7be72b4eef80daacdcdfde82a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.653468 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c29b11192604ac18c21b7bdc0238fb3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:36.672544 12229 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 consensus state has no leader
I20260810 23:32:36.684868 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874ca0c3586b41a8a3074d8988813633" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:36.679977 13279 tablet_service.cc:1467] Processing CreateTablet for tablet 4d697e7533af4fc0a15cb41f3fcce4db (DEFAULT_TABLE table=default.Table_26bf2c7d86704b5b981ecebb4e90ba75 [id=cefe7b97c1db42ef8712bf3c7a359af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.690231 14006 raft_consensus.cc:482] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:32:36.691387 12700 leader_election.cc:341] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:36.690932 14006 raft_consensus.cc:504] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:36.697134 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d697e7533af4fc0a15cb41f3fcce4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.702272 14006 leader_election.cc:288] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:36.724519 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f791b2df0524997ab1c53a9e5cb39ca" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:36.725518 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 consensus state has no leader
I20260810 23:32:36.725977 13336 raft_consensus.cc:2379] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate 191d120f4c7f44fc8475305a6d2e75f3 in this term.
I20260810 23:32:36.721311 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874ca0c3586b41a8a3074d8988813633" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:36.738754 12958 leader_election.cc:341] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 1f791b2df0524997ab1c53a9e5cb39ca
I20260810 23:32:36.739400 12958 leader_election.cc:302] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:36.741928 14006 raft_consensus.cc:3037] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:36.698864 13943 raft_consensus.cc:482] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:36.768000 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d596299c82b4ff1ac151d68b97ff55b with delete_type TABLET_DATA_DELETED (replaced by tablet 4cdec13ee70d402fa3a08119e7bf1313) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:36.785159 13277 tablet_service.cc:1467] Processing CreateTablet for tablet 4b71d565669f40d8a6d06ab267069430 (DEFAULT_TABLE table=default.Table_93dd9072dddd48ec859e82f2c6e0244c [id=24e3b49b5b8f4bcfabc485ddf55ee01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.792797 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45f9dfd03764d7fb1b6fb5e28b66bd3" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:36.795249 12698 leader_election.cc:341] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:36.796078 12698 leader_election.cc:302] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:36.797989 14030 raft_consensus.cc:2726] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:36.801106 13291 tablet_service.cc:1467] Processing CreateTablet for tablet fd4f55ef709d473f89a16d32160eb61b (DEFAULT_TABLE table=default.Table_addbc3e258944d0cb5d2e7fb11214412 [id=d4e32fc5299547268eb0c93e2d3c00e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.802940 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4f55ef709d473f89a16d32160eb61b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:36.806349 12698 leader_election.cc:341] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:36.807193 12698 leader_election.cc:302] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:36.808923 14030 raft_consensus.cc:2726] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:36.815267 14030 raft_consensus.cc:482] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:36.815971 14030 raft_consensus.cc:504] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:36.821481 14030 leader_election.cc:288] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:36.821451 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840400119d20424ea0f03f2bff550d44" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:36.823570 12700 leader_election.cc:341] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:36.867192 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 234539us
I20260810 23:32:36.869650 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet b261673faa964b65af3576c89e302864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:36.871642 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 840400119d20424ea0f03f2bff550d44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 840400119d20424ea0f03f2bff550d44 already in progress: creating tablet
W20260810 23:32:36.872742 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 3d596299c82b4ff1ac151d68b97ff55b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d596299c82b4ff1ac151d68b97ff55b already in progress: creating tablet
I20260810 23:32:36.873733 13974 ts_tablet_manager.cc:1918] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:36.874205 13974 log.cc:1196] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/0c0d6cd2ad1d4a65929b0d856a171592
I20260810 23:32:36.875075 13974 ts_tablet_manager.cc:1939] T 0c0d6cd2ad1d4a65929b0d856a171592 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:36.762087 14006 raft_consensus.cc:2726] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:36.880247 13665 tablet_bootstrap.cc:654] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:36.766366 13943 raft_consensus.cc:504] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:36.907335 12240 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca0dd19112504f8ca57f64dcf0bd50a8 has no consensus state
I20260810 23:32:36.908735 12240 master_service.cc:946] ReplaceTablet: received request to replace tablet 81e26fe4d42445279f12bb17b2841d08 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:36.911079 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840400119d20424ea0f03f2bff550d44" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:36.900141 13276 tablet_service.cc:1467] Processing CreateTablet for tablet ab8d987109ba40198645881b449cb972 (DEFAULT_TABLE table=default.Table_d8a0e3e655b9417d9df3ff6d82b76f15 [id=d0b86eafe9544cd98d8fce8ece3b5c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.762997 13278 tablet_service.cc:1467] Processing CreateTablet for tablet 423e8d583d704fa9834026ebcd57e17e (DEFAULT_TABLE table=default.Table_514d038dba3a4fc399b97e7edc254ad9 [id=53909ddaaede4deb97cec7ad02020c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.871409 13691 tablet_bootstrap.cc:654] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:36.936432 12236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_7c223ce60be04a498d08155325785819" } modify_external_catalogs: true
I20260810 23:32:36.937068 12236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_7c223ce60be04a498d08155325785819" } modify_external_catalogs: true
W20260810 23:32:36.938449 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet b261673faa964b65af3576c89e302864 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b261673faa964b65af3576c89e302864 already in progress: creating tablet
I20260810 23:32:36.940374 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_241642ff9438427d8938dd88c19ba164"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:36.944242 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet b261673faa964b65af3576c89e302864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:36.945617 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_241642ff9438427d8938dd88c19ba164 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:36.949122 13275 tablet_service.cc:1467] Processing CreateTablet for tablet 9c29b11192604ac18c21b7bdc0238fb3 (DEFAULT_TABLE table=default.Table_dd2cd30e5e2349de9180be01866dc513 [id=8754da7be72b4eef80daacdcdfde82a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:36.950951 13277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b71d565669f40d8a6d06ab267069430. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:36.952610 12698 leader_election.cc:341] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:36.953125 12698 leader_election.cc:302] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:36.954324 13764 raft_consensus.cc:2726] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:36.954820 13935 ts_tablet_manager.cc:1918] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:36.955276 13935 log.cc:1196] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/81e26fe4d42445279f12bb17b2841d08
I20260810 23:32:36.956219 13935 ts_tablet_manager.cc:1939] T 81e26fe4d42445279f12bb17b2841d08 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:36.961103 13943 leader_election.cc:288] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:36.937814 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e26fe4d42445279f12bb17b2841d08" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:36.962649 13062 raft_consensus.cc:2454] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:36.963912 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 423e8d583d704fa9834026ebcd57e17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.969138 13275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c29b11192604ac18c21b7bdc0238fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:36.957463 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 70) took 1656 ms. Trace:
I20260810 23:32:36.972517 13935 rpcz_store.cc:276] 0810 23:32:35.301399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.301687 (+ 288us) service_pool.cc:227] Handling call
0810 23:32:35.999190 (+697503us) ts_tablet_manager.cc:1145] Deleting tablet 81e26fe4d42445279f12bb17b2841d08
0810 23:32:35.999311 (+ 121us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:36.530744 (+531433us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:36.954295 (+423551us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:36.957370 (+ 3075us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":696530}
I20260810 23:32:36.985849 13935 tablet_replica.cc:330] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:36.987823 13935 raft_consensus.cc:2229] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:36.988696 13514 tablet_bootstrap.cc:492] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:36.989239 13514 ts_tablet_manager.cc:1397] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.690s user 0.032s sys 0.039s
I20260810 23:32:36.990948 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8986e59b2314bd69b9b1de36f1f2ba7" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:36.992095 12806 raft_consensus.cc:2454] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:36.992895 13935 raft_consensus.cc:2258] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:36.994402 13514 raft_consensus.cc:348] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:36.995427 13514 raft_consensus.cc:374] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:36.995697 13514 raft_consensus.cc:729] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:36.996448 13935 ts_tablet_manager.cc:1905] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:36.996476 13514 consensus_queue.cc:260] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:37.003690 13514 ts_tablet_manager.cc:1428] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
W20260810 23:32:37.003643 13280 consensus_meta.cc:287] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent flushing consensus metadata: real 0.666s user 0.009s sys 0.004s
I20260810 23:32:37.005462 13280 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 80) took 1055 ms. Trace:
I20260810 23:32:37.006089 13280 rpcz_store.cc:276] 0810 23:32:35.950070 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.950274 (+ 204us) service_pool.cc:227] Handling call
0810 23:32:35.952447 (+ 2173us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:35.952568 (+ 121us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:36.336636 (+384068us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:37.004477 (+667841us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:37.005351 (+ 874us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 23:32:36.949103 13021 consensus_meta.cc:287] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.501s user 0.004s sys 0.008s
I20260810 23:32:37.015048 13514 tablet_bootstrap.cc:492] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:37.024852 14030 raft_consensus.cc:482] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:37.028066 13935 ts_tablet_manager.cc:1918] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:37.028888 13935 log.cc:1196] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/8b1f7c16b6bc4dc0bac80ac55332f056
I20260810 23:32:37.030398 13935 ts_tablet_manager.cc:1939] T 8b1f7c16b6bc4dc0bac80ac55332f056 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:37.025625 14030 raft_consensus.cc:504] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:37.031857 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 71) took 1729 ms. Trace:
I20260810 23:32:37.032462 13935 rpcz_store.cc:276] 0810 23:32:35.302198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.303534 (+ 1336us) service_pool.cc:227] Handling call
0810 23:32:36.985363 (+1681829us) ts_tablet_manager.cc:1145] Deleting tablet 8b1f7c16b6bc4dc0bac80ac55332f056
0810 23:32:36.985492 (+ 129us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:37.012458 (+ 26966us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:37.027633 (+ 15175us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:37.031740 (+ 4107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1681095}
I20260810 23:32:37.034819 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 8b1f7c16b6bc4dc0bac80ac55332f056 (table default.Table_e6944762a73d4c80ae0866fad8945b41 [id=50e2197e7c13449caeebf39a42ba347c]) successfully deleted
I20260810 23:32:37.035490 13935 tablet_replica.cc:330] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:37.037235 13935 raft_consensus.cc:2229] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:37.038199 13935 raft_consensus.cc:2258] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:37.040746 14030 leader_election.cc:288] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:37.039644 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ba75f2c2b54ec1968db8feeacf8383" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:37.043051 13935 ts_tablet_manager.cc:1905] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:37.070785 13549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:37.071758 13549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:37.078581 13286 consensus_meta.cc:287] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent flushing consensus metadata: real 0.789s user 0.011s sys 0.002s
W20260810 23:32:37.092646 13282 consensus_meta.cc:287] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent flushing consensus metadata: real 0.545s user 0.009s sys 0.008s
W20260810 23:32:37.107439 13549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:37.112306 13549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:37.126647 13691 tablet_bootstrap.cc:492] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:37.127197 13691 ts_tablet_manager.cc:1397] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.550s user 0.009s sys 0.009s
W20260810 23:32:37.129129 12700 leader_election.cc:341] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:37.129844 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 58236.1us
W20260810 23:32:37.131664 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 192438us
I20260810 23:32:37.134702 12228 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_36d49207cc33452db87dd140a5193dc6" } new_table_name: "default.Table_4f38414090ae4a5cac18c58cccb303e4" modify_external_catalogs: true
I20260810 23:32:37.136449 13691 raft_consensus.cc:348] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:37.137246 13691 raft_consensus.cc:374] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:37.137527 13691 raft_consensus.cc:729] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:37.138376 13691 consensus_queue.cc:260] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:37.141260 13691 ts_tablet_manager.cc:1428] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.014s user 0.004s sys 0.006s
I20260810 23:32:37.148679 12235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet db363eb0f98d460eb6f5a577233d8ac0
I20260810 23:32:37.155057 13021 tablet_service.cc:1514] Processing DeleteTablet for tablet db363eb0f98d460eb6f5a577233d8ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:36 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:37.159287 13514 tablet_bootstrap.cc:654] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:37.159021 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet db363eb0f98d460eb6f5a577233d8ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:36 UTC) from {username='slave'} at 127.0.0.1:47546
W20260810 23:32:37.159785 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet db363eb0f98d460eb6f5a577233d8ac0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db363eb0f98d460eb6f5a577233d8ac0 already in progress: creating tablet
I20260810 23:32:37.162287 13691 tablet_bootstrap.cc:492] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
W20260810 23:32:37.091694 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 96343.5us
I20260810 23:32:37.193280 13286 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 81) took 1235 ms. Trace:
I20260810 23:32:37.194236 13286 rpcz_store.cc:276] 0810 23:32:35.957488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.957717 (+ 229us) service_pool.cc:227] Handling call
0810 23:32:35.959244 (+ 1527us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:35.959353 (+ 109us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:36.288865 (+329512us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:37.079669 (+790804us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:37.193143 (+113474us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:37.199631 13691 tablet_bootstrap.cc:654] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:37.223107 13691 tablet_bootstrap.cc:492] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:37.224749 13691 ts_tablet_manager.cc:1397] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.063s user 0.016s sys 0.009s
I20260810 23:32:37.230355 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab8d987109ba40198645881b449cb972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:37.231626 13691 raft_consensus.cc:348] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:37.232437 13691 raft_consensus.cc:374] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:37.232733 13691 raft_consensus.cc:729] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:37.238513 13691 consensus_queue.cc:260] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:37.242178 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_d8373b61098245bcaab43448a20578f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:37.245100 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8373b61098245bcaab43448a20578f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:37.258440 13282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 82) took 1193 ms. Trace:
I20260810 23:32:37.266070 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ba75f2c2b54ec1968db8feeacf8383" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:37.267970 13691 ts_tablet_manager.cc:1428] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.041s user 0.006s sys 0.001s
I20260810 23:32:37.271418 13691 tablet_bootstrap.cc:492] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:37.284507 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:37.285969 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_41341e2688304faa88c0ccc969e5faa7 [id=b809029bf6cc48ab917e48bbf2109ae1]). Expected version 1 got 0
W20260810 23:32:37.286218 14035 log.cc:924] Time spent T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3: Append to log took a long time: real 0.309s user 0.002s sys 0.001s
I20260810 23:32:37.287164 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 8d2f8688fc6046d58009800c5e288302 (table default.Table_783bf3bd81bb4ec294327f9ed5411afb [id=1f49f73dd9d34e968defd3a8ade21d22]). Expected version 1 got 0
I20260810 23:32:37.291599 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet dce0ec83ed8943a0b0788505bb75d1df (table default.Table_94e24b4b27d047a99df427c453fe1752 [id=e462d54230344223a457f708501d2d30]). Expected version 3 got 0
I20260810 23:32:37.295012 13691 tablet_bootstrap.cc:654] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:37.339886 12224 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_41341e2688304faa88c0ccc969e5faa7" } new_table_name: "default.Table_75ca982c15e34c8d9603f100dff5b392" modify_external_catalogs: true
I20260810 23:32:37.284724 13213 leader_election.cc:302] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
W20260810 23:32:37.345613 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 63978.9us
I20260810 23:32:37.349169 13866 raft_consensus.cc:2781] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:37.349715 13866 raft_consensus.cc:482] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:37.350260 13866 raft_consensus.cc:3037] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:37.357028 14041 consensus_queue.cc:1035] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260810 23:32:37.354539 13868 consensus_queue.cc:1035] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260810 23:32:37.390380 13514 tablet_bootstrap.cc:492] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:37.390995 13514 ts_tablet_manager.cc:1397] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.376s user 0.008s sys 0.012s
W20260810 23:32:37.395162 12698 leader_election.cc:341] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:37.395767 12698 leader_election.cc:302] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:37.398838 14030 raft_consensus.cc:2726] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:37.199455 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 77) took 1858 ms. Trace:
I20260810 23:32:37.260120 13282 rpcz_store.cc:276] 0810 23:32:36.065209 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.068261 (+ 3052us) service_pool.cc:227] Handling call
0810 23:32:36.070911 (+ 2650us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:36.071099 (+ 188us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:36.546357 (+475258us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:37.180417 (+634060us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:37.257735 (+ 77318us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b5800001898. stack: 00007f8764d74033 00007f8764d73f30 00007f87648f5534 00000000004db2b4 00000000004db249 00007f8764beaf11 00007f8764d6e851 00007f8764d6ee82 00007f876dc2e8b9 00007f876cc625d8 00007f876cd26063 00007f876cd21e2f 00007f876cd225ed 00007f876ccd3869 00007f876b3c5c44 00007f876b3c5bd2
0810 23:32:37.258335 (+ 600us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25252608}
I20260810 23:32:37.433615 13665 tablet_bootstrap.cc:492] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:37.434526 13665 ts_tablet_manager.cc:1397] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 1.223s user 0.019s sys 0.005s
I20260810 23:32:37.403478 13974 rpcz_store.cc:276] 0810 23:32:35.356891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.358311 (+ 1420us) service_pool.cc:227] Handling call
0810 23:32:35.843624 (+485313us) ts_tablet_manager.cc:1145] Deleting tablet 0c0d6cd2ad1d4a65929b0d856a171592
0810 23:32:35.843678 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:36.522974 (+679296us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:36.873396 (+350422us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:37.199326 (+325930us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":482277}
I20260810 23:32:37.456507 13974 tablet_replica.cc:330] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:37.457755 13974 raft_consensus.cc:2229] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:37.461274 13935 ts_tablet_manager.cc:1918] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:37.461771 13935 log.cc:1196] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/79da9583d3524347a9be09e35affaaad
I20260810 23:32:37.462523 13935 ts_tablet_manager.cc:1939] T 79da9583d3524347a9be09e35affaaad P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:37.477095 14006 raft_consensus.cc:482] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:37.477761 14006 raft_consensus.cc:504] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:37.457348 13514 raft_consensus.cc:348] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:37.479784 13514 raft_consensus.cc:374] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:37.480173 13514 raft_consensus.cc:729] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:37.480803 14006 leader_election.cc:288] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:37.482775 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2f8688fc6046d58009800c5e288302" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:37.483462 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2f8688fc6046d58009800c5e288302" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:37.486043 12958 leader_election.cc:302] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:37.487352 14006 raft_consensus.cc:2726] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:37.490619 13691 tablet_bootstrap.cc:492] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:37.491185 13691 ts_tablet_manager.cc:1397] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.220s user 0.064s sys 0.069s
I20260810 23:32:37.494760 13691 raft_consensus.cc:348] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:37.495656 13691 raft_consensus.cc:374] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:37.495972 13691 raft_consensus.cc:729] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:37.496872 13691 consensus_queue.cc:260] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:37.492620 13514 consensus_queue.cc:260] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:37.504398 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8986e59b2314bd69b9b1de36f1f2ba7" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:37.509016 12228 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42517f11422b42c39db282e2c5f584a3 consensus state has no leader
I20260810 23:32:37.514223 13691 ts_tablet_manager.cc:1428] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.023s user 0.004s sys 0.004s
I20260810 23:32:37.519850 13691 tablet_bootstrap.cc:492] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:37.525810 13514 ts_tablet_manager.cc:1428] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.134s user 0.010s sys 0.000s
I20260810 23:32:37.526324 13866 raft_consensus.cc:504] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:37.529615 13866 leader_election.cc:288] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:37.527887 13059 raft_consensus.cc:3037] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:37.530380 13691 tablet_bootstrap.cc:654] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:37.531271 13212 leader_election.cc:302] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:37.537348 14041 raft_consensus.cc:2781] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:37.537887 14041 raft_consensus.cc:482] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:37.538341 14041 raft_consensus.cc:3037] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:37.453161 13665 raft_consensus.cc:348] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:37.560674 13665 raft_consensus.cc:374] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:37.561126 13665 raft_consensus.cc:729] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:37.561975 13665 consensus_queue.cc:260] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:37.454113 13213 consensus_peers.cc:487] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b1f7c16b6bc4dc0bac80ac55332f056. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:37.563645 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 118305us
I20260810 23:32:37.568423 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8986e59b2314bd69b9b1de36f1f2ba7" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:37.569458 12806 raft_consensus.cc:3037] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:37.573585 13514 tablet_bootstrap.cc:492] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:32:37.576977 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 45804us
I20260810 23:32:37.461405 13283 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 84) took 1112 ms. Trace:
I20260810 23:32:37.581723 14041 raft_consensus.cc:504] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:37.582120 13283 rpcz_store.cc:276] 0810 23:32:36.348895 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.539651 (+190756us) service_pool.cc:227] Handling call
0810 23:32:36.552589 (+ 12938us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:36.552860 (+ 271us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:37.077327 (+524467us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:37.460800 (+383473us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:37.461325 (+ 525us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:37.596824 14041 leader_election.cc:288] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:37.599467 13691 tablet_bootstrap.cc:492] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:37.599929 13691 ts_tablet_manager.cc:1397] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.080s user 0.019s sys 0.010s
I20260810 23:32:37.602615 13691 raft_consensus.cc:348] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:37.603479 13691 raft_consensus.cc:374] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:37.603729 13691 raft_consensus.cc:729] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:37.604547 13691 consensus_queue.cc:260] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:37.613435 13514 tablet_bootstrap.cc:654] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:37.623142 13691 ts_tablet_manager.cc:1428] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20260810 23:32:37.648645 13514 tablet_bootstrap.cc:492] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:37.648815 13691 tablet_bootstrap.cc:492] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:37.649616 13514 ts_tablet_manager.cc:1397] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.076s user 0.019s sys 0.004s
I20260810 23:32:37.661298 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 73) took 2357 ms. Trace:
I20260810 23:32:37.661998 13935 rpcz_store.cc:276] 0810 23:32:35.303621 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.303876 (+ 255us) service_pool.cc:227] Handling call
0810 23:32:37.035098 (+1731222us) ts_tablet_manager.cc:1145] Deleting tablet 79da9583d3524347a9be09e35affaaad
0810 23:32:37.035194 (+ 96us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:37.169695 (+134501us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:37.460961 (+291266us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:37.661197 (+200236us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1730057}
I20260810 23:32:37.665001 13691 tablet_bootstrap.cc:654] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:37.668978 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 90333.9us
W20260810 23:32:37.669993 13212 leader_election.cc:341] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:37.676401 13866 consensus_queue.cc:1035] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:32:37.680187 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 98422.4us
I20260810 23:32:37.680683 13974 raft_consensus.cc:2258] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:37.683753 13062 raft_consensus.cc:3037] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:37.687964 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 0c0d6cd2ad1d4a65929b0d856a171592 (table default.Table_9e4b92615b33447baec585c92237ff05 [id=287124fe670e4041a492d9a4d4524980]) successfully deleted
I20260810 23:32:37.689805 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8986e59b2314bd69b9b1de36f1f2ba7" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:37.690843 13058 raft_consensus.cc:3037] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:37.692945 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e26fe4d42445279f12bb17b2841d08" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:37.693994 13060 raft_consensus.cc:3037] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:37.697242 13691 tablet_bootstrap.cc:492] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:37.697841 13691 ts_tablet_manager.cc:1397] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.013s
I20260810 23:32:37.703632 13691 raft_consensus.cc:348] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:37.704519 13691 raft_consensus.cc:374] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:37.704769 13691 raft_consensus.cc:729] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:37.707063 13974 ts_tablet_manager.cc:1905] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:37.711982 13868 consensus_queue.cc:1035] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260810 23:32:37.719218 13213 leader_election.cc:341] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 81e26fe4d42445279f12bb17b2841d08
I20260810 23:32:37.721297 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet db363eb0f98d460eb6f5a577233d8ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:36 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:37.750864 12214 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 42517f11422b42c39db282e2c5f584a3 (table default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 42517f11422b42c39db282e2c5f584a3 consensus state has no leader
I20260810 23:32:37.658998 13514 raft_consensus.cc:348] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:37.768492 13514 raft_consensus.cc:374] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:37.769053 13514 raft_consensus.cc:729] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:37.762933 13691 consensus_queue.cc:260] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:37.770336 13514 consensus_queue.cc:260] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:37.777192 13514 ts_tablet_manager.cc:1428] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.127s user 0.009s sys 0.002s
I20260810 23:32:37.778247 13514 tablet_bootstrap.cc:492] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:37.782562 13691 ts_tablet_manager.cc:1428] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.084s user 0.007s sys 0.005s
I20260810 23:32:37.783746 13691 tablet_bootstrap.cc:492] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:37.784540 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 79da9583d3524347a9be09e35affaaad (table default.Table_275ea19d22624a12a6b3b2c8e985c4af [id=5b4476e15a7e410599b70f84ab267de9]) successfully deleted
I20260810 23:32:37.787226 13935 tablet_replica.cc:330] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:37.792629 13935 raft_consensus.cc:2229] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:37.695858 12806 raft_consensus.cc:2454] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:37.828948 13691 tablet_bootstrap.cc:654] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:37.843279 13935 raft_consensus.cc:2258] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:37.860992 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_32d9165e885d480881ced6b879bcd759"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:37.862789 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32d9165e885d480881ced6b879bcd759 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:37.878298 13665 ts_tablet_manager.cc:1428] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.443s user 0.007s sys 0.001s
I20260810 23:32:37.879650 13665 tablet_bootstrap.cc:492] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:37.889106 13868 consensus_queue.cc:1035] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:37.897114 13277 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 89) took 1178 ms. Trace:
I20260810 23:32:37.897737 13277 rpcz_store.cc:276] 0810 23:32:36.718973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.763476 (+ 44503us) service_pool.cc:227] Handling call
0810 23:32:36.876987 (+113511us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:36.884376 (+ 7389us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:37.465960 (+581584us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:37.896164 (+430204us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:37.897010 (+ 846us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:37.903136 13935 ts_tablet_manager.cc:1905] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:37.940449 13691 tablet_bootstrap.cc:492] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:37.940986 13691 ts_tablet_manager.cc:1397] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.157s user 0.021s sys 0.005s
I20260810 23:32:37.941642 13021 tablet_service.cc:1467] Processing CreateTablet for tablet ca0dd19112504f8ca57f64dcf0bd50a8 (DEFAULT_TABLE table=default.Table_1e5f07908d444cf9af2cae2ea83ae7ae [id=a591a34da3514b87aa852684669a399f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:37.943480 13213 leader_election.cc:302] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:37.944178 13021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0dd19112504f8ca57f64dcf0bd50a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:37.944420 13691 raft_consensus.cc:348] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:37.945148 13691 raft_consensus.cc:374] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:37.945398 13691 raft_consensus.cc:729] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:37.946277 13943 raft_consensus.cc:2781] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:37.928982 13060 raft_consensus.cc:2454] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:37.946904 13943 raft_consensus.cc:686] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:37.949172 13943 consensus_queue.cc:237] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:37.948892 13058 raft_consensus.cc:2454] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:37.952961 13017 tablet_service.cc:1467] Processing CreateTablet for tablet fd9d90e1ceab443dbec12b2c81b975e8 (DEFAULT_TABLE table=default.Table_3e86c4915aac433bb544495b6bd22c1c [id=aef6fd3aebac448e96d5942fd12836ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:37.954295 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd9d90e1ceab443dbec12b2c81b975e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:37.954234 12769 tablet_service.cc:1467] Processing CreateTablet for tablet ca0dd19112504f8ca57f64dcf0bd50a8 (DEFAULT_TABLE table=default.Table_1e5f07908d444cf9af2cae2ea83ae7ae [id=a591a34da3514b87aa852684669a399f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:37.954967 13691 consensus_queue.cc:260] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:37.957430 12229 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:37.957978 13691 ts_tablet_manager.cc:1428] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260810 23:32:37.958384 12229 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 874ca0c3586b41a8a3074d8988813633 (table default.Table_b57f0f6e94384d22af5d3bf7817d5044 [id=4bb9a6fbe6e644c583a468af86950cef]). Expected version 1 got 0
I20260810 23:32:37.959447 12229 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 8d2f8688fc6046d58009800c5e288302 (table default.Table_783bf3bd81bb4ec294327f9ed5411afb [id=1f49f73dd9d34e968defd3a8ade21d22]). Expected version 1 got 0
I20260810 23:32:37.966957 13691 tablet_bootstrap.cc:492] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:37.977921 12229 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]). Expected version 1 got 0
I20260810 23:32:37.985249 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_41341e2688304faa88c0ccc969e5faa7 [id=b809029bf6cc48ab917e48bbf2109ae1]). Expected version 1 got 0
I20260810 23:32:37.986457 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:37.991410 13691 tablet_bootstrap.cc:654] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:37.994299 13012 tablet_service.cc:1467] Processing CreateTablet for tablet e6b8bf38da5647889756935404caad75 (DEFAULT_TABLE table=default.Table_f34283baaf0d495b8a5b076e45292979 [id=3f2a52251dba491c8f6acfa4da4960e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:37.995756 13012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b8bf38da5647889756935404caad75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:37.996632 12764 tablet_service.cc:1467] Processing CreateTablet for tablet fd9d90e1ceab443dbec12b2c81b975e8 (DEFAULT_TABLE table=default.Table_3e86c4915aac433bb544495b6bd22c1c [id=aef6fd3aebac448e96d5942fd12836ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:37.996992 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_34f40e29ad7a4068943dc4c724bce3b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:37.999214 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34f40e29ad7a4068943dc4c724bce3b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:38.007679 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd9d90e1ceab443dbec12b2c81b975e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.019380 13275 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 93) took 1137 ms. Trace:
I20260810 23:32:38.019556 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0dd19112504f8ca57f64dcf0bd50a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.019964 13275 rpcz_store.cc:276] 0810 23:32:36.882092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.932397 (+ 50305us) service_pool.cc:227] Handling call
0810 23:32:36.960266 (+ 27869us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:36.960376 (+ 110us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:37.635350 (+674974us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:38.018422 (+383072us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:38.019269 (+ 847us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:38.021525 12240 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:38.022521 12240 catalog_manager.cc:5495] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } }
I20260810 23:32:38.025939 12240 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_41341e2688304faa88c0ccc969e5faa7 [id=b809029bf6cc48ab917e48bbf2109ae1]). Expected version 1 got 0
I20260810 23:32:38.027745 12240 catalog_manager.cc:5495] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } }
I20260810 23:32:38.030761 12240 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 8d2f8688fc6046d58009800c5e288302 (table default.Table_783bf3bd81bb4ec294327f9ed5411afb [id=1f49f73dd9d34e968defd3a8ade21d22]). Expected version 1 got 0
I20260810 23:32:38.033422 12235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_783bf3bd81bb4ec294327f9ed5411afb" } modify_external_catalogs: true
I20260810 23:32:38.034053 12235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_783bf3bd81bb4ec294327f9ed5411afb" } modify_external_catalogs: true
I20260810 23:32:38.039033 12766 tablet_service.cc:1467] Processing CreateTablet for tablet e6b8bf38da5647889756935404caad75 (DEFAULT_TABLE table=default.Table_f34283baaf0d495b8a5b076e45292979 [id=3f2a52251dba491c8f6acfa4da4960e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.041934 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b8bf38da5647889756935404caad75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.045853 12761 tablet_service.cc:1467] Processing CreateTablet for tablet c5794637d5dd43f2b05edd53865f4e99 (DEFAULT_TABLE table=default.Table_a0956ecd557b42e292fde54f05842c47 [id=339a844c37ee460cb948d7e58264e5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.048319 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5794637d5dd43f2b05edd53865f4e99. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:38.007251 13278 consensus_meta.cc:287] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent flushing consensus metadata: real 0.520s user 0.009s sys 0.005s
I20260810 23:32:38.069281 13514 tablet_bootstrap.cc:700] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent opening tablet: real 0.187s user 0.003s sys 0.000s
I20260810 23:32:38.070183 13514 tablet_bootstrap.cc:654] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:38.071048 14006 raft_consensus.cc:482] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:38.072047 14006 raft_consensus.cc:504] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:38.077621 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b29cda94584badb1225477ec2bd1bc" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:38.079128 13278 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 88) took 1398 ms. Trace:
W20260810 23:32:38.085143 12952 leader_election.cc:341] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:38.083294 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b29cda94584badb1225477ec2bd1bc" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:38.086781 12802 raft_consensus.cc:2454] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:38.089233 12958 leader_election.cc:302] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:38.067178 13010 tablet_service.cc:1467] Processing CreateTablet for tablet c5794637d5dd43f2b05edd53865f4e99 (DEFAULT_TABLE table=default.Table_a0956ecd557b42e292fde54f05842c47 [id=339a844c37ee460cb948d7e58264e5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.095228 13010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5794637d5dd43f2b05edd53865f4e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.119784 12767 tablet_service.cc:1467] Processing CreateTablet for tablet 10033af79e854103bc47a046d2219ca0 (DEFAULT_TABLE table=default.Table_620c572fc75b44fe8ca892789875c1f7 [id=cf58351bb8594bdbb92168f093ec91a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.123108 12767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10033af79e854103bc47a046d2219ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.142442 12772 tablet_service.cc:1467] Processing CreateTablet for tablet 3fef127263f1470ea1056a454971e465 (DEFAULT_TABLE table=default.Table_023c0d2da11a419f8c2eb9dc2a064cee [id=1386100c13dc4d54975429c0dffbef72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.146425 12772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fef127263f1470ea1056a454971e465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.110766 14006 leader_election.cc:288] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:38.150277 14006 raft_consensus.cc:2781] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:38.150945 14006 raft_consensus.cc:482] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:38.151546 14006 raft_consensus.cc:3037] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:38.196230 14093 raft_consensus.cc:482] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:38.197273 14093 raft_consensus.cc:504] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:38.216461 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 127423us
I20260810 23:32:38.219388 13014 tablet_service.cc:1467] Processing CreateTablet for tablet 10033af79e854103bc47a046d2219ca0 (DEFAULT_TABLE table=default.Table_620c572fc75b44fe8ca892789875c1f7 [id=cf58351bb8594bdbb92168f093ec91a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.221424 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10033af79e854103bc47a046d2219ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.229928 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 3fef127263f1470ea1056a454971e465 (DEFAULT_TABLE table=default.Table_023c0d2da11a419f8c2eb9dc2a064cee [id=1386100c13dc4d54975429c0dffbef72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.231210 13514 tablet_bootstrap.cc:492] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:38.233379 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fef127263f1470ea1056a454971e465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.235231 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 92b81e7056a441d2854ba5ab6b4c5433 (DEFAULT_TABLE table=default.Table_cdd5220b787349159091ea6dc3359c9a [id=d8d3c66123cd4516a59a0ab8ae4d47b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.237272 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b81e7056a441d2854ba5ab6b4c5433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.231832 13514 ts_tablet_manager.cc:1397] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.454s user 0.012s sys 0.006s
I20260810 23:32:38.246387 12228 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:38.253175 12228 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_41341e2688304faa88c0ccc969e5faa7 [id=b809029bf6cc48ab917e48bbf2109ae1]). Expected version 1 got 0
I20260810 23:32:38.251976 12770 tablet_service.cc:1467] Processing CreateTablet for tablet c01ede4b52c3498a924340b39c3c2435 (DEFAULT_TABLE table=default.Table_69fa732a88234f859e1cbb8d706d7c76 [id=ae7b5faec4c34d35981e0a1d464202ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.265749 12236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet dd83bbe7a1664c278098fa66b9c6895e
I20260810 23:32:38.267704 12236 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 164) took 1418 ms. Trace:
I20260810 23:32:38.273686 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 92b81e7056a441d2854ba5ab6b4c5433 (DEFAULT_TABLE table=default.Table_cdd5220b787349159091ea6dc3359c9a [id=d8d3c66123cd4516a59a0ab8ae4d47b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.278483 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c01ede4b52c3498a924340b39c3c2435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.280083 14093 leader_election.cc:288] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:38.281010 14093 raft_consensus.cc:482] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:38.281714 14093 raft_consensus.cc:504] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:38.282847 13514 raft_consensus.cc:348] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:38.285599 13514 raft_consensus.cc:374] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:38.285954 13514 raft_consensus.cc:729] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:38.286232 12235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 8d2f8688fc6046d58009800c5e288302
I20260810 23:32:38.287236 13514 consensus_queue.cc:260] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:38.290568 12760 tablet_service.cc:1467] Processing CreateTablet for tablet c3d4c879ca6a402dbafe91940e7b367a (DEFAULT_TABLE table=default.Table_6e13e5c0a9da4cc5a1b85b4a247f8a3f [id=820d73489eed49d8a72357b3ac31300d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.292891 12763 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdec13ee70d402fa3a08119e7bf1313 (DEFAULT_TABLE table=default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.294323 13691 tablet_bootstrap.cc:492] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:38.294873 13691 ts_tablet_manager.cc:1397] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.336s user 0.014s sys 0.011s
I20260810 23:32:38.301187 12760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d4c879ca6a402dbafe91940e7b367a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.307160 14006 raft_consensus.cc:504] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:38.311259 14006 leader_election.cc:288] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:38.303074 12237 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_bacbea393b664097a0b52af2ce5292de [id=71c9424f327040a68889fc257935c535] alter complete (version 1)
I20260810 23:32:38.272010 12236 rpcz_store.cc:276] 0810 23:32:36.849477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.914366 (+ 64889us) service_pool.cc:227] Handling call
0810 23:32:36.936296 (+ 21930us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7466 00007f0ce6e28b34 00007f0ce6e28ac2 00007f0ce6e289f1 00007f0ce6e2896c 00007f0ce6e27c22 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:36.937507 (+ 1211us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:36.938096 (+ 589us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:36.938254 (+ 158us) catalog_manager.cc:2761] Locking tablets
0810 23:32:37.261236 (+322982us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:37.732497 (+471261us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:37.732835 (+ 338us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:38.265598 (+532763us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:38.267618 (+ 2020us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:37.504515 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:37.505355 (+ 840us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:37.505385 (+ 30us) write_op.cc:435] Acquired schema lock
0810 23:32:37.505402 (+ 17us) tablet.cc:660] Decoding operations
0810 23:32:37.505813 (+ 411us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:37.505860 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:37.505886 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:37.506358 (+ 472us) tablet.cc:699] Row locks acquired
0810 23:32:37.506375 (+ 17us) write_op.cc:260] PREPARE: finished
0810 23:32:37.506546 (+ 171us) write_op.cc:270] Start()
0810 23:32:37.506665 (+ 119us) write_op.cc:276] Timestamp: P: 1786404757506399 usec, L: 0
0810 23:32:37.506687 (+ 22us) op_driver.cc:331] REPLICATION: starting
0810 23:32:37.507384 (+ 697us) log.cc:842] Serialized 768 byte log entry
0810 23:32:37.714420 (+207036us) op_driver.cc:444] REPLICATION: finished
0810 23:32:37.717268 (+ 2848us) write_op.cc:301] APPLY: starting
0810 23:32:37.717400 (+ 132us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:37.717609 (+ 209us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:37.717805 (+ 196us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:37.718286 (+ 481us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:37.718653 (+ 367us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:37.718732 (+ 79us) write_op.cc:312] APPLY: finished
0810 23:32:37.719068 (+ 336us) log.cc:842] Serialized 29 byte log entry
0810 23:32:37.719299 (+ 231us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:37.719452 (+ 153us) write_op.cc:454] Released schema lock
0810 23:32:37.719518 (+ 66us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:37.743490 (+ 23972us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:37.743542 (+ 52us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":48056704,"child_traces":[["op",{"apply.queue_time_us":2671,"apply.run_cpu_time_us":3051,"apply.run_wall_time_us":26583,"mutex_wait_us":11712,"num_ops":2,"prepare.queue_time_us":240767,"prepare.run_cpu_time_us":3559,"prepare.run_wall_time_us":234356,"raft.queue_time_us":11959,"raft.run_cpu_time_us":1284,"raft.run_wall_time_us":1257,"replication_time_us":207645,"thread_start_us":142294,"threads_started":1}]]}
I20260810 23:32:38.325402 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0d9227d8574de8bd73041f3e8043a0 (DEFAULT_TABLE table=default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:38.307126 12226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 17) took 2898 ms (client timeout 1000 ms). Trace:
I20260810 23:32:38.327533 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet dd83bbe7a1664c278098fa66b9c6895e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:36 UTC) from {username='slave'} at 127.0.0.1:47546
W20260810 23:32:38.328398 12226 rpcz_store.cc:269] 0810 23:32:35.408672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.566536 (+157864us) service_pool.cc:227] Handling call
0810 23:32:38.307032 (+2740496us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":8}
W20260810 23:32:38.256114 13276 consensus_meta.cc:287] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent flushing consensus metadata: real 0.608s user 0.006s sys 0.011s
I20260810 23:32:38.309099 12224 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 167) took 1194 ms. Trace:
I20260810 23:32:38.337194 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdec13ee70d402fa3a08119e7bf1313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.337633 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_32d7e9316d054deca361c6e3d9b5e1f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:38.132469 13866 consensus_queue.cc:1035] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.131s
I20260810 23:32:38.341368 13276 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 92) took 1483 ms. Trace:
I20260810 23:32:38.341910 13276 rpcz_store.cc:276] 0810 23:32:36.857765 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.876447 (+ 18682us) service_pool.cc:227] Handling call
0810 23:32:37.228057 (+351610us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b5800001898. stack: 00007f8764d74033 00007f8764d73f30 00007f87648f5534 00000000004db2b4 00000000004db249 00007f8764beaf11 00007f8764d6e851 00007f8764d6ee82 00007f876dc2e8b9 00007f876cc625d8 00007f876cd222a0 00007f876ccd3869 00007f876b3c5c44 00007f876b3c5bd2 00007f876b3c5b01 00007f876b3c5a7c
0810 23:32:37.228122 (+ 65us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:37.228258 (+ 136us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:37.647124 (+418866us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:38.340377 (+693253us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:38.341250 (+ 873us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2755328}
I20260810 23:32:38.329550 12224 rpcz_store.cc:276] 0810 23:32:37.114324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:37.306529 (+192205us) service_pool.cc:227] Handling call
0810 23:32:37.339430 (+ 32901us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e77d6 00007f0ce6e2d634 00007f0ce6e2d5c2 00007f0ce6e2d4f1 00007f0ce6e2d46c 00007f0ce6e2c722 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:37.341911 (+ 2481us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:37.343192 (+ 1281us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:37.345223 (+ 2031us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:37.345606 (+ 383us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:37.907277 (+561671us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:38.308981 (+401704us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:37.739088 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:37.739620 (+ 532us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:37.739647 (+ 27us) write_op.cc:435] Acquired schema lock
0810 23:32:37.739661 (+ 14us) tablet.cc:660] Decoding operations
0810 23:32:37.740053 (+ 392us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:37.740103 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:37.740126 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:37.740316 (+ 190us) tablet.cc:699] Row locks acquired
0810 23:32:37.740331 (+ 15us) write_op.cc:260] PREPARE: finished
0810 23:32:37.740496 (+ 165us) write_op.cc:270] Start()
0810 23:32:37.740602 (+ 106us) write_op.cc:276] Timestamp: P: 1786404757740334 usec, L: 0
0810 23:32:37.740619 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:32:37.858288 (+117669us) log.cc:842] Serialized 405 byte log entry
0810 23:32:37.859742 (+ 1454us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:37.904677 (+ 44935us) op_driver.cc:444] REPLICATION: finished
0810 23:32:37.905245 (+ 568us) write_op.cc:301] APPLY: starting
0810 23:32:37.905334 (+ 89us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:37.905507 (+ 173us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:37.905558 (+ 51us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:37.905817 (+ 259us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:37.906063 (+ 246us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:37.906162 (+ 99us) write_op.cc:312] APPLY: finished
0810 23:32:37.906487 (+ 325us) log.cc:842] Serialized 23 byte log entry
0810 23:32:37.906735 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:37.906851 (+ 116us) write_op.cc:454] Released schema lock
0810 23:32:37.906908 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:37.907845 (+ 937us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:37.907900 (+ 55us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":72439808,"child_traces":[["op",{"apply.queue_time_us":310,"apply.run_cpu_time_us":2368,"apply.run_wall_time_us":2983,"num_ops":1,"prepare.queue_time_us":352260,"prepare.run_cpu_time_us":5148,"prepare.run_wall_time_us":120933,"raft.queue_time_us":9504,"raft.run_cpu_time_us":845,"raft.run_wall_time_us":845,"replication_time_us":163979,"spinlock_wait_cycles":269100928}]]}
I20260810 23:32:38.351231 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d2f8688fc6046d58009800c5e288302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:38 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:38.352404 13514 ts_tablet_manager.cc:1428] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.114s user 0.006s sys 0.003s
I20260810 23:32:38.079735 13278 rpcz_store.cc:276] 0810 23:32:36.681155 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.703469 (+ 22314us) service_pool.cc:227] Handling call
0810 23:32:36.949523 (+246054us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:36.958800 (+ 9277us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:37.486775 (+527975us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:38.078147 (+591372us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:38.079009 (+ 862us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:38.357287 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 0d1b15dd545c4598aa5c7a9dc53618ec (DEFAULT_TABLE table=default.Table_2c509bc8585b4ca68502ce3897fa18fa [id=15edc7d801a94b40b8ab324e25d530e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.359956 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d1b15dd545c4598aa5c7a9dc53618ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.341670 14092 raft_consensus.cc:482] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:38.371182 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0d9227d8574de8bd73041f3e8043a0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:38.340332 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32d7e9316d054deca361c6e3d9b5e1f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:38.370285 14092 raft_consensus.cc:504] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:38.372121 13764 raft_consensus.cc:482] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:38.373363 12224 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_cdd5220b787349159091ea6dc3359c9a" } new_table_name: "default.Table_9a86e4c2641044419dffb5dbef58e581" modify_external_catalogs: true
W20260810 23:32:38.374644 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 381797us
I20260810 23:32:38.375360 14092 leader_election.cc:288] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:38.363082 14093 leader_election.cc:288] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:38.343667 13691 raft_consensus.cc:348] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:38.375730 13212 leader_election.cc:302] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:38.376920 13691 raft_consensus.cc:374] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:38.377290 13691 raft_consensus.cc:729] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:38.378151 13691 consensus_queue.cc:260] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:38.379555 13943 raft_consensus.cc:2781] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:38.384274 13691 ts_tablet_manager.cc:1428] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.089s user 0.011s sys 0.002s
I20260810 23:32:38.385281 13691 tablet_bootstrap.cc:492] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
W20260810 23:32:38.327000 12237 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 24) took 3047 ms (client timeout 1000 ms). Trace:
I20260810 23:32:38.406350 13943 raft_consensus.cc:686] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
W20260810 23:32:38.405293 12237 rpcz_store.cc:269] 0810 23:32:35.280047 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.282284 (+ 2237us) service_pool.cc:227] Handling call
0810 23:32:35.333055 (+ 50771us) spinlock_profiling.cc:243] Waited 47.5 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:38.326910 (+2993855us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:36.464985 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:36.465591 (+ 606us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:36.465631 (+ 40us) write_op.cc:435] Acquired schema lock
0810 23:32:36.465657 (+ 26us) tablet.cc:660] Decoding operations
0810 23:32:36.466064 (+ 407us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:36.466116 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:36.466157 (+ 41us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:36.466336 (+ 179us) tablet.cc:699] Row locks acquired
0810 23:32:36.466353 (+ 17us) write_op.cc:260] PREPARE: finished
0810 23:32:36.466548 (+ 195us) write_op.cc:270] Start()
0810 23:32:36.466658 (+ 110us) write_op.cc:276] Timestamp: P: 1786404756466493 usec, L: 0
0810 23:32:36.466682 (+ 24us) op_driver.cc:331] REPLICATION: starting
0810 23:32:36.467332 (+ 650us) log.cc:842] Serialized 356 byte log entry
0810 23:32:36.664267 (+196935us) op_driver.cc:444] REPLICATION: finished
0810 23:32:36.665639 (+ 1372us) write_op.cc:301] APPLY: starting
0810 23:32:36.665742 (+ 103us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:36.665953 (+ 211us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:36.666162 (+ 209us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:36.666491 (+ 329us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:36.666705 (+ 214us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:36.666809 (+ 104us) write_op.cc:312] APPLY: finished
0810 23:32:36.667159 (+ 350us) log.cc:842] Serialized 23 byte log entry
0810 23:32:36.667363 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:36.667463 (+ 100us) write_op.cc:454] Released schema lock
0810 23:32:36.667526 (+ 63us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:36.667848 (+ 322us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:36.667911 (+ 63us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":10765,"reported_tablets":5,"spinlock_wait_cycles":109282688,"child_traces":[["op",{"apply.queue_time_us":1086,"apply.run_cpu_time_us":2617,"apply.run_wall_time_us":5542,"num_ops":1,"prepare.queue_time_us":476395,"prepare.run_cpu_time_us":2885,"prepare.run_wall_time_us":31389,"replication_time_us":197489}]]}
I20260810 23:32:38.414785 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_d6f432a771c544d6acfb28ec150340da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:38.424398 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6f432a771c544d6acfb28ec150340da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:38.425616 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 46514.6us
I20260810 23:32:38.316819 12236 master_service.cc:946] ReplaceTablet: received request to replace tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:38.465236 13665 tablet_bootstrap.cc:654] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:38.471109 13943 consensus_queue.cc:237] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:38.481066 13514 tablet_bootstrap.cc:492] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:38.482899 13278 tablet_service.cc:1467] Processing CreateTablet for tablet ca0dd19112504f8ca57f64dcf0bd50a8 (DEFAULT_TABLE table=default.Table_1e5f07908d444cf9af2cae2ea83ae7ae [id=a591a34da3514b87aa852684669a399f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.493381 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0dd19112504f8ca57f64dcf0bd50a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.504626 13276 tablet_service.cc:1467] Processing CreateTablet for tablet fd9d90e1ceab443dbec12b2c81b975e8 (DEFAULT_TABLE table=default.Table_3e86c4915aac433bb544495b6bd22c1c [id=aef6fd3aebac448e96d5942fd12836ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.506242 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd9d90e1ceab443dbec12b2c81b975e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.514360 13275 tablet_service.cc:1467] Processing CreateTablet for tablet e6b8bf38da5647889756935404caad75 (DEFAULT_TABLE table=default.Table_f34283baaf0d495b8a5b076e45292979 [id=3f2a52251dba491c8f6acfa4da4960e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.515866 13275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b8bf38da5647889756935404caad75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.532387 13277 tablet_service.cc:1467] Processing CreateTablet for tablet c5794637d5dd43f2b05edd53865f4e99 (DEFAULT_TABLE table=default.Table_a0956ecd557b42e292fde54f05842c47 [id=339a844c37ee460cb948d7e58264e5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.533874 13277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5794637d5dd43f2b05edd53865f4e99. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:38.538746 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 85253.7us
I20260810 23:32:38.546340 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 10033af79e854103bc47a046d2219ca0 (DEFAULT_TABLE table=default.Table_620c572fc75b44fe8ca892789875c1f7 [id=cf58351bb8594bdbb92168f093ec91a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.566558 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10033af79e854103bc47a046d2219ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.573120 14107 raft_consensus.cc:482] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:38.573830 14107 raft_consensus.cc:504] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:38.589401 13279 tablet_service.cc:1467] Processing CreateTablet for tablet 3fef127263f1470ea1056a454971e465 (DEFAULT_TABLE table=default.Table_023c0d2da11a419f8c2eb9dc2a064cee [id=1386100c13dc4d54975429c0dffbef72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.590934 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fef127263f1470ea1056a454971e465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.603854 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 92b81e7056a441d2854ba5ab6b4c5433 (DEFAULT_TABLE table=default.Table_cdd5220b787349159091ea6dc3359c9a [id=d8d3c66123cd4516a59a0ab8ae4d47b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.605492 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b81e7056a441d2854ba5ab6b4c5433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.605688 12226 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 172) took 1094 ms. Trace:
I20260810 23:32:38.606060 12226 rpcz_store.cc:276] 0810 23:32:37.511668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.336218 (+824550us) service_pool.cc:227] Handling call
0810 23:32:38.372324 (+ 36106us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:38.404789 (+ 32465us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:38.604439 (+199650us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:38.604479 (+ 40us) write_op.cc:454] Released schema lock
0810 23:32:38.604791 (+ 312us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:38.605224 (+ 433us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:38.605594 (+ 370us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:38.542488 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:38.542970 (+ 482us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:38.543003 (+ 33us) write_op.cc:435] Acquired schema lock
0810 23:32:38.543017 (+ 14us) tablet.cc:660] Decoding operations
0810 23:32:38.543381 (+ 364us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:38.543429 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:38.543454 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:38.543697 (+ 243us) tablet.cc:699] Row locks acquired
0810 23:32:38.543715 (+ 18us) write_op.cc:260] PREPARE: finished
0810 23:32:38.543866 (+ 151us) write_op.cc:270] Start()
0810 23:32:38.543970 (+ 104us) write_op.cc:276] Timestamp: P: 1786404758543755 usec, L: 0
0810 23:32:38.543990 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:32:38.545234 (+ 1244us) log.cc:842] Serialized 433 byte log entry
0810 23:32:38.561186 (+ 15952us) op_driver.cc:444] REPLICATION: finished
0810 23:32:38.561650 (+ 464us) write_op.cc:301] APPLY: starting
0810 23:32:38.561737 (+ 87us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:38.561922 (+ 185us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:38.562103 (+ 181us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:38.562693 (+ 590us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:38.562929 (+ 236us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:38.563001 (+ 72us) write_op.cc:312] APPLY: finished
0810 23:32:38.563286 (+ 285us) log.cc:842] Serialized 29 byte log entry
0810 23:32:38.563521 (+ 235us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:38.563633 (+ 112us) write_op.cc:454] Released schema lock
0810 23:32:38.563680 (+ 47us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":225,"apply.run_cpu_time_us":2317,"apply.run_wall_time_us":2315,"mutex_wait_us":135909,"num_ops":2,"prepare.queue_time_us":136043,"prepare.run_cpu_time_us":2819,"prepare.run_wall_time_us":3224,"raft.queue_time_us":4214,"raft.run_cpu_time_us":1182,"raft.run_wall_time_us":1183,"replication_time_us":17081}]]}
I20260810 23:32:38.623134 13278 tablet_service.cc:1467] Processing CreateTablet for tablet c01ede4b52c3498a924340b39c3c2435 (DEFAULT_TABLE table=default.Table_69fa732a88234f859e1cbb8d706d7c76 [id=ae7b5faec4c34d35981e0a1d464202ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.630460 13549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:41081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.226.190:43533,127.11.226.189:42101,127.11.226.188:45477
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.189:42101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.226.189
--webserver_port=37775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:32:38.644039 13549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:38.647529 13549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:32:38.662179 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdec13ee70d402fa3a08119e7bf1313 (DEFAULT_TABLE table=default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.663128 13935 ts_tablet_manager.cc:1918] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:38.664700 13291 tablet_service.cc:1467] Processing CreateTablet for tablet c3d4c879ca6a402dbafe91940e7b367a (DEFAULT_TABLE table=default.Table_6e13e5c0a9da4cc5a1b85b4a247f8a3f [id=820d73489eed49d8a72357b3ac31300d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.668305 13273 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0d6cd2ad1d4a65929b0d856a171592 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:38.668047 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0d9227d8574de8bd73041f3e8043a0 (DEFAULT_TABLE table=default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.669337 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet dd83bbe7a1664c278098fa66b9c6895e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:36 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:38.669654 13273 tablet_service.cc:1514] Processing DeleteTablet for tablet 79da9583d3524347a9be09e35affaaad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:38.670346 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d2f8688fc6046d58009800c5e288302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:38 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:38.713445 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124259164adf404c9cbe5f4e0d5e16c0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:38.736163 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0d9227d8574de8bd73041f3e8043a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.741123 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d4c879ca6a402dbafe91940e7b367a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.741909 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdec13ee70d402fa3a08119e7bf1313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.761888 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c01ede4b52c3498a924340b39c3c2435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.784474 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b29cda94584badb1225477ec2bd1bc" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:38.784953 13935 log.cc:1196] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/0c0d6cd2ad1d4a65929b0d856a171592
I20260810 23:32:38.785997 13935 ts_tablet_manager.cc:1939] T 0c0d6cd2ad1d4a65929b0d856a171592 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:38.416296 13764 raft_consensus.cc:504] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:38.788493 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d596299c82b4ff1ac151d68b97ff55b" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:38.789372 13764 leader_election.cc:288] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:38.665017 14107 leader_election.cc:288] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:38.674837 13514 tablet_bootstrap.cc:654] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:38.700341 12236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet c45f9dfd03764d7fb1b6fb5e28b66bd3
I20260810 23:32:38.795578 12236 catalog_manager.cc:6565] ReplaceTablet: tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 deleted and replaced by tablet 4d1b9d20d09c4d1ba9a4b10959807e50
I20260810 23:32:38.312511 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b81e7056a441d2854ba5ab6b4c5433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.796268 12236 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 171) took 1419 ms. Trace:
W20260810 23:32:38.798812 12952 leader_election.cc:341] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:38.796600 12236 rpcz_store.cc:276] 0810 23:32:37.376639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.316549 (+939910us) service_pool.cc:227] Handling call
0810 23:32:38.796189 (+479640us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:38.679937 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:38.680437 (+ 500us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:38.680468 (+ 31us) write_op.cc:435] Acquired schema lock
0810 23:32:38.680482 (+ 14us) tablet.cc:660] Decoding operations
0810 23:32:38.680876 (+ 394us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:38.680928 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:38.680954 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:38.681193 (+ 239us) tablet.cc:699] Row locks acquired
0810 23:32:38.681207 (+ 14us) write_op.cc:260] PREPARE: finished
0810 23:32:38.681465 (+ 258us) write_op.cc:270] Start()
0810 23:32:38.681577 (+ 112us) write_op.cc:276] Timestamp: P: 1786404758681332 usec, L: 0
0810 23:32:38.681594 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:32:38.682217 (+ 623us) log.cc:842] Serialized 631 byte log entry
0810 23:32:38.696539 (+ 14322us) op_driver.cc:444] REPLICATION: finished
0810 23:32:38.696968 (+ 429us) write_op.cc:301] APPLY: starting
0810 23:32:38.697171 (+ 203us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:38.697297 (+ 126us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:38.697407 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:38.697810 (+ 403us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:38.698054 (+ 244us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:38.698106 (+ 52us) write_op.cc:312] APPLY: finished
0810 23:32:38.698324 (+ 218us) log.cc:842] Serialized 29 byte log entry
0810 23:32:38.698500 (+ 176us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:38.698584 (+ 84us) write_op.cc:454] Released schema lock
0810 23:32:38.698624 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:38.698830 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:38.698854 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":241,"apply.run_cpu_time_us":1963,"apply.run_wall_time_us":2079,"num_ops":2,"prepare.queue_time_us":152,"prepare.run_cpu_time_us":2818,"prepare.run_wall_time_us":99400,"raft.queue_time_us":743,"raft.run_cpu_time_us":816,"raft.run_wall_time_us":819,"replication_time_us":14839}]]}
W20260810 23:32:38.726794 12224 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92b81e7056a441d2854ba5ab6b4c5433 consensus state has no leader
W20260810 23:32:38.777176 12214 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 92b81e7056a441d2854ba5ab6b4c5433 (table default.Table_9a86e4c2641044419dffb5dbef58e581 [id=d8d3c66123cd4516a59a0ab8ae4d47b9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92b81e7056a441d2854ba5ab6b4c5433 consensus state has no leader
I20260810 23:32:38.801126 12224 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 173) took 1085 ms. Trace:
W20260810 23:32:38.800371 12952 leader_election.cc:341] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:38.808204 12698 leader_election.cc:341] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:38.811106 12952 leader_election.cc:341] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 585baf44aca142199fbc268ff3bcd8f4
I20260810 23:32:38.819646 13012 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdec13ee70d402fa3a08119e7bf1313 (DEFAULT_TABLE table=default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.821600 13012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdec13ee70d402fa3a08119e7bf1313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.823341 13021 tablet_service.cc:1467] Processing CreateTablet for tablet c3d4c879ca6a402dbafe91940e7b367a (DEFAULT_TABLE table=default.Table_6e13e5c0a9da4cc5a1b85b4a247f8a3f [id=820d73489eed49d8a72357b3ac31300d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.825140 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_73f2b1b207fb45f2adecbd37412c5e6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:38.825852 13017 tablet_service.cc:1467] Processing CreateTablet for tablet c01ede4b52c3498a924340b39c3c2435 (DEFAULT_TABLE table=default.Table_69fa732a88234f859e1cbb8d706d7c76 [id=ae7b5faec4c34d35981e0a1d464202ef]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:38.827385 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73f2b1b207fb45f2adecbd37412c5e6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:38.837373 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b29cda94584badb1225477ec2bd1bc" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:38.837051 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d596299c82b4ff1ac151d68b97ff55b" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:38.825603 13514 tablet_bootstrap.cc:492] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:38.838119 12806 raft_consensus.cc:3037] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:38.838191 12802 raft_consensus.cc:2454] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:38.838493 13514 ts_tablet_manager.cc:1397] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.358s user 0.017s sys 0.019s
I20260810 23:32:38.839509 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45f9dfd03764d7fb1b6fb5e28b66bd3" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:38.840216 12803 raft_consensus.cc:2454] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:38.841391 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0d9227d8574de8bd73041f3e8043a0 (DEFAULT_TABLE table=default.Table_085f24edadf34deea7a3d9356d00c7b1 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.841361 13514 raft_consensus.cc:348] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:38.842101 13691 tablet_bootstrap.cc:654] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:38.842202 13514 raft_consensus.cc:374] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:38.842473 13514 raft_consensus.cc:729] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
W20260810 23:32:38.845849 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 44332.2us
I20260810 23:32:38.846012 12760 tablet_service.cc:1514] Processing DeleteTablet for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d1b9d20d09c4d1ba9a4b10959807e50) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:38.853415 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c01ede4b52c3498a924340b39c3c2435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.856746 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0d9227d8574de8bd73041f3e8043a0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:38.860260 12764 consensus_meta.cc:287] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.525s user 0.012s sys 0.011s
I20260810 23:32:38.860693 13021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d4c879ca6a402dbafe91940e7b367a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.860818 12806 raft_consensus.cc:2454] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:38.864161 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_d942bf134f9849718bb9eb12c24806a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:38.801592 12224 rpcz_store.cc:276] 0810 23:32:37.716084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.372889 (+656805us) service_pool.cc:227] Handling call
0810 23:32:38.436300 (+ 63411us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:38.437377 (+ 1077us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:38.438192 (+ 815us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:38.631042 (+192850us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:38.726048 (+ 95006us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:38.801055 (+ 75007us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:38.632646 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:38.633047 (+ 401us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:38.633071 (+ 24us) write_op.cc:435] Acquired schema lock
0810 23:32:38.633121 (+ 50us) tablet.cc:660] Decoding operations
0810 23:32:38.633434 (+ 313us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:38.633475 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:38.633505 (+ 30us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:38.633669 (+ 164us) tablet.cc:699] Row locks acquired
0810 23:32:38.633682 (+ 13us) write_op.cc:260] PREPARE: finished
0810 23:32:38.633804 (+ 122us) write_op.cc:270] Start()
0810 23:32:38.633902 (+ 98us) write_op.cc:276] Timestamp: P: 1786404758633692 usec, L: 0
0810 23:32:38.633920 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 23:32:38.634837 (+ 917us) log.cc:842] Serialized 405 byte log entry
0810 23:32:38.648590 (+ 13753us) op_driver.cc:444] REPLICATION: finished
0810 23:32:38.649021 (+ 431us) write_op.cc:301] APPLY: starting
0810 23:32:38.649086 (+ 65us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:38.649284 (+ 198us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:38.649402 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:38.649606 (+ 204us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:38.649826 (+ 220us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:38.649877 (+ 51us) write_op.cc:312] APPLY: finished
0810 23:32:38.650115 (+ 238us) log.cc:842] Serialized 23 byte log entry
0810 23:32:38.650328 (+ 213us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:38.650408 (+ 80us) write_op.cc:454] Released schema lock
0810 23:32:38.650449 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:38.650687 (+ 238us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:38.650715 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":198,"apply.run_cpu_time_us":1797,"apply.run_wall_time_us":1892,"num_ops":1,"prepare.queue_time_us":104,"prepare.run_cpu_time_us":2210,"prepare.run_wall_time_us":8012,"replication_time_us":14569,"spinlock_wait_cycles":738688}]]}
I20260810 23:32:38.869668 12224 master_service.cc:946] ReplaceTablet: received request to replace tablet 13ad5d63a2104995990a86a000963cdc from {username='slave'} at 127.0.0.1:34742
W20260810 23:32:38.871130 12767 consensus_meta.cc:287] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.521s user 0.020s sys 0.002s
I20260810 23:32:38.873620 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 72) took 3570 ms. Trace:
I20260810 23:32:38.878777 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124259164adf404c9cbe5f4e0d5e16c0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:38.896355 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f791b2df0524997ab1c53a9e5cb39ca with delete_type TABLET_DATA_DELETED (replaced by tablet c892e6bc117a4fff917342b8eefaf74d) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:38.784152 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45f9dfd03764d7fb1b6fb5e28b66bd3" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:38.902968 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:38.903529 13026 tablet_service.cc:1514] Processing DeleteTablet for tablet dd83bbe7a1664c278098fa66b9c6895e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:36 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:38.905504 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d2f8688fc6046d58009800c5e288302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:38 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:38.906710 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0d6cd2ad1d4a65929b0d856a171592 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:38.908557 13007 tablet_service.cc:1514] Processing DeleteTablet for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d1b9d20d09c4d1ba9a4b10959807e50) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:38.907712 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet 585baf44aca142199fbc268ff3bcd8f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:27 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:38.912190 13008 tablet_service.cc:1467] Processing CreateTablet for tablet 0d1b15dd545c4598aa5c7a9dc53618ec (DEFAULT_TABLE table=default.Table_2c509bc8585b4ca68502ce3897fa18fa [id=15edc7d801a94b40b8ab324e25d530e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.913849 13008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d1b15dd545c4598aa5c7a9dc53618ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.915516 12958 leader_election.cc:302] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:38.917199 12958 leader_election.cc:302] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:38.917753 13273 tablet_service.cc:1467] Processing CreateTablet for tablet 0d1b15dd545c4598aa5c7a9dc53618ec (DEFAULT_TABLE table=default.Table_2c509bc8585b4ca68502ce3897fa18fa [id=15edc7d801a94b40b8ab324e25d530e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:38.921474 13273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d1b15dd545c4598aa5c7a9dc53618ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:38.923604 13935 rpcz_store.cc:276] 0810 23:32:35.302867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.303076 (+ 209us) service_pool.cc:227] Handling call
0810 23:32:37.786867 (+2483791us) ts_tablet_manager.cc:1145] Deleting tablet 0c0d6cd2ad1d4a65929b0d856a171592
0810 23:32:37.786984 (+ 117us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:38.277835 (+490851us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:38.662791 (+384956us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:38.873531 (+210740us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":84637,"tablet-delete.queue_time_us":2480690}
I20260810 23:32:38.925254 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d1b9d20d09c4d1ba9a4b10959807e50) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:38.925791 12236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_dd2cd30e5e2349de9180be01866dc513" } modify_external_catalogs: true
I20260810 23:32:38.928536 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046f1a7983ec40489edeaba4d2595b1c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:38.935204 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d942bf134f9849718bb9eb12c24806a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:38.843158 13514 consensus_queue.cc:260] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:38.939707 12236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_dd2cd30e5e2349de9180be01866dc513" } modify_external_catalogs: true
W20260810 23:32:38.953377 12952 leader_election.cc:341] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:38.954381 13935 tablet_replica.cc:330] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:38.960628 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 0c0d6cd2ad1d4a65929b0d856a171592 (table default.Table_9e4b92615b33447baec585c92237ff05 [id=287124fe670e4041a492d9a4d4524980]) successfully deleted
I20260810 23:32:38.939395 13866 consensus_queue.cc:1035] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260810 23:32:38.968703 13935 raft_consensus.cc:2229] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:38.974016 14116 raft_consensus.cc:2781] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:38.974614 14116 raft_consensus.cc:482] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:38.975076 14116 raft_consensus.cc:3037] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:38.961905 13514 ts_tablet_manager.cc:1428] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.123s user 0.007s sys 0.000s
I20260810 23:32:38.979399 13514 tablet_bootstrap.cc:492] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:38.982847 13935 raft_consensus.cc:2258] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:38.982990 14123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:38.919937 13943 raft_consensus.cc:482] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:38.992775 13943 raft_consensus.cc:504] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:39.019999 13335 raft_consensus.cc:2454] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:32:39.023893 12698 leader_election.cc:302] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:39.025731 14107 raft_consensus.cc:2781] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:39.026259 14107 raft_consensus.cc:482] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:39.027022 14107 raft_consensus.cc:3037] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:39.031612 12959 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.690s user 0.011s sys 0.002s
I20260810 23:32:39.005144 13935 ts_tablet_manager.cc:1905] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:39.048848 13943 leader_election.cc:288] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:39.069936 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046f1a7983ec40489edeaba4d2595b1c" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:39.084951 13665 tablet_bootstrap.cc:492] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:39.085637 13665 ts_tablet_manager.cc:1397] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 1.206s user 0.020s sys 0.016s
I20260810 23:32:39.091744 13764 raft_consensus.cc:482] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:39.092705 13764 raft_consensus.cc:504] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:39.098323 13764 leader_election.cc:288] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:39.108402 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd77526f1db4455291fde8a001092a09" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:39.112185 12698 leader_election.cc:341] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:39.117511 13514 tablet_bootstrap.cc:654] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:39.153048 13974 ts_tablet_manager.cc:1918] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:39.154423 13974 log.cc:1196] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/8b1f7c16b6bc4dc0bac80ac55332f056
W20260810 23:32:39.028940 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 111822us
W20260810 23:32:39.161525 12958 leader_election.cc:341] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 585baf44aca142199fbc268ff3bcd8f4
I20260810 23:32:39.162284 12958 leader_election.cc:302] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:39.162758 12237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_6e13e5c0a9da4cc5a1b85b4a247f8a3f" } modify_external_catalogs: true
I20260810 23:32:39.163784 12237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_6e13e5c0a9da4cc5a1b85b4a247f8a3f" } modify_external_catalogs: true
I20260810 23:32:39.169442 14116 raft_consensus.cc:504] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:39.091388 13665 raft_consensus.cc:348] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:39.191457 13665 raft_consensus.cc:374] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:39.191876 13665 raft_consensus.cc:729] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:39.205278 13665 consensus_queue.cc:260] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:39.208505 13665 ts_tablet_manager.cc:1428] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.122s user 0.007s sys 0.004s
I20260810 23:32:39.210559 13974 ts_tablet_manager.cc:1939] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:39.225348 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 78) took 3856 ms. Trace:
I20260810 23:32:39.232174 13974 rpcz_store.cc:276] 0810 23:32:35.369115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.372804 (+ 3689us) service_pool.cc:227] Handling call
0810 23:32:37.456236 (+2083432us) ts_tablet_manager.cc:1145] Deleting tablet 8b1f7c16b6bc4dc0bac80ac55332f056
0810 23:32:37.456311 (+ 75us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:37.586291 (+129980us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b58000902b0. stack: 00007f8764d74033 00007f8764d73f30 00007f87648f5534 00000000004db2b4 00000000004db249 00007f876bc80471 00007f876bc76f63 00007f876bcdaaba 00007f876bd03375 00007f876c11926c 00007f876cd26cfa 00007f876cd56e10 00007f876cd3de0d 00007f876cd3ddb9 00007f876cd3dd49 00007f876cd3dd11
0810 23:32:38.468888 (+882597us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:39.152659 (+683771us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:39.225227 (+ 72568us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":92961,"spinlock_wait_cycles":289565440,"tablet-delete.queue_time_us":2082567}
I20260810 23:32:39.234840 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 86) took 2653 ms. Trace:
I20260810 23:32:39.235266 13974 rpcz_store.cc:276] 0810 23:32:36.581936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.582823 (+ 887us) service_pool.cc:227] Handling call
0810 23:32:39.234303 (+2651480us) ts_tablet_manager.cc:1145] Deleting tablet 840400119d20424ea0f03f2bff550d44
0810 23:32:39.234353 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:39.234762 (+ 409us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2593338}
I20260810 23:32:39.244874 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 8b1f7c16b6bc4dc0bac80ac55332f056 (table default.Table_e6944762a73d4c80ae0866fad8945b41 [id=50e2197e7c13449caeebf39a42ba347c]) successfully deleted
I20260810 23:32:39.248767 13014 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 93) took 1043 ms. Trace:
I20260810 23:32:39.249250 13014 rpcz_store.cc:276] 0810 23:32:38.217883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.218061 (+ 178us) service_pool.cc:227] Handling call
0810 23:32:38.220567 (+ 2506us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:38.220730 (+ 163us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:38.892065 (+671335us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:39.027118 (+135053us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:39.248406 (+221288us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:39.252931 14124 raft_consensus.cc:2781] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:39.253463 14124 raft_consensus.cc:482] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:39.253911 14124 raft_consensus.cc:3037] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:39.258386 13273 tablet_service.cc:1467] Processing CreateTablet for tablet f1e8311c941b43058384cee6d8263dc1 (DEFAULT_TABLE table=default.Table_d8373b61098245bcaab43448a20578f6 [id=7934cc21dac24c74848a668abe1dd606]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.263029 13273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e8311c941b43058384cee6d8263dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.267166 12757 tablet_service.cc:1467] Processing CreateTablet for tablet f1e8311c941b43058384cee6d8263dc1 (DEFAULT_TABLE table=default.Table_d8373b61098245bcaab43448a20578f6 [id=7934cc21dac24c74848a668abe1dd606]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.269275 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e8311c941b43058384cee6d8263dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.270992 14116 leader_election.cc:288] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:39.272631 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d596299c82b4ff1ac151d68b97ff55b" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:39.274353 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0136a1e890436fae572b46e64750e7 (DEFAULT_TABLE table=default.Table_32d9165e885d480881ced6b879bcd759 [id=bece3c4a5e8e4dc4bb23129238561321]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:39.274713 12952 leader_election.cc:341] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:39.275797 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0136a1e890436fae572b46e64750e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.267913 14107 raft_consensus.cc:504] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:39.282087 12803 raft_consensus.cc:2379] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in current term 1: Already voted for candidate 191d120f4c7f44fc8475305a6d2e75f3 in this term.
I20260810 23:32:39.284195 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 90) took 2517 ms. Trace:
W20260810 23:32:39.285311 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 840400119d20424ea0f03f2bff550d44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 840400119d20424ea0f03f2bff550d44 already in progress: creating tablet
I20260810 23:32:39.289237 12763 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0136a1e890436fae572b46e64750e7 (DEFAULT_TABLE table=default.Table_32d9165e885d480881ced6b879bcd759 [id=bece3c4a5e8e4dc4bb23129238561321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.291049 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0136a1e890436fae572b46e64750e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.292685 14136 raft_consensus.cc:482] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:39.293098 13974 rpcz_store.cc:276] 0810 23:32:36.766875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.767780 (+ 905us) service_pool.cc:227] Handling call
0810 23:32:39.283555 (+2515775us) ts_tablet_manager.cc:1145] Deleting tablet 3d596299c82b4ff1ac151d68b97ff55b
0810 23:32:39.283643 (+ 88us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:39.284092 (+ 449us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2515013}
I20260810 23:32:39.296226 14107 leader_election.cc:288] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:39.301473 12224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 13ad5d63a2104995990a86a000963cdc
W20260810 23:32:39.301777 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 3d596299c82b4ff1ac151d68b97ff55b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d596299c82b4ff1ac151d68b97ff55b already in progress: creating tablet
I20260810 23:32:39.303048 12224 catalog_manager.cc:6565] ReplaceTablet: tablet 13ad5d63a2104995990a86a000963cdc deleted and replaced by tablet 8fb2ee1c9b41478da001aa67d7a7ad3c
I20260810 23:32:39.304308 12235 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:39.212983 13665 tablet_bootstrap.cc:492] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:39.309087 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ad5d63a2104995990a86a000963cdc with delete_type TABLET_DATA_DELETED (replaced by tablet 8fb2ee1c9b41478da001aa67d7a7ad3c) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:39.311872 12770 tablet_service.cc:1467] Processing CreateTablet for tablet cd24471dd52149d395808493ad9fa68c (DEFAULT_TABLE table=default.Table_241642ff9438427d8938dd88c19ba164 [id=2696163e827d4abca86cb624622710bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.301661 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046f1a7983ec40489edeaba4d2595b1c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:39.312376 12764 tablet_service.cc:1467] Processing CreateTablet for tablet 4bce222cd5ab41fdb12fff3fe3dfbd3c (DEFAULT_TABLE table=default.Table_34f40e29ad7a4068943dc4c724bce3b9 [id=3d5ee1f5b27f4707b79ed3fef7b8ffa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.312798 13335 raft_consensus.cc:3037] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:39.313194 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd24471dd52149d395808493ad9fa68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.313918 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bce222cd5ab41fdb12fff3fe3dfbd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.305760 13290 tablet_service.cc:1467] Processing CreateTablet for tablet cd24471dd52149d395808493ad9fa68c (DEFAULT_TABLE table=default.Table_241642ff9438427d8938dd88c19ba164 [id=2696163e827d4abca86cb624622710bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.323926 13335 raft_consensus.cc:2454] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:39.328260 12698 leader_election.cc:302] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:39.300376 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 94) took 2377 ms. Trace:
I20260810 23:32:39.316582 12224 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_0d38f0b73fd746b4a487e7b1600eae77" } new_table_name: "default.Table_dfb60e6e40874a5da0c56f6926ce1f63" modify_external_catalogs: true
I20260810 23:32:39.330935 13974 rpcz_store.cc:276] 0810 23:32:36.922982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.941044 (+ 18062us) service_pool.cc:227] Handling call
0810 23:32:39.299725 (+2358681us) ts_tablet_manager.cc:1145] Deleting tablet b261673faa964b65af3576c89e302864
0810 23:32:39.299797 (+ 72us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:39.300279 (+ 482us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2354882}
I20260810 23:32:39.330490 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd24471dd52149d395808493ad9fa68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.335114 12235 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 874ca0c3586b41a8a3074d8988813633 (table default.Table_b57f0f6e94384d22af5d3bf7817d5044 [id=4bb9a6fbe6e644c583a468af86950cef]). Expected version 1 got 0
I20260810 23:32:39.338168 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 95) took 1617 ms. Trace:
W20260810 23:32:39.342386 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet b261673faa964b65af3576c89e302864 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b261673faa964b65af3576c89e302864 already in progress: creating tablet
I20260810 23:32:39.346251 14101 raft_consensus.cc:2781] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:39.342097 13278 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ad5d63a2104995990a86a000963cdc with delete_type TABLET_DATA_DELETED (replaced by tablet 8fb2ee1c9b41478da001aa67d7a7ad3c) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:39.346951 14101 raft_consensus.cc:686] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Becoming Leader. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Running, Role: LEADER
I20260810 23:32:39.343648 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 4bce222cd5ab41fdb12fff3fe3dfbd3c (DEFAULT_TABLE table=default.Table_34f40e29ad7a4068943dc4c724bce3b9 [id=3d5ee1f5b27f4707b79ed3fef7b8ffa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.348238 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bce222cd5ab41fdb12fff3fe3dfbd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.349826 14101 consensus_queue.cc:237] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:39.344961 13274 tablet_service.cc:1467] Processing CreateTablet for tablet f47b8b57d5b44a018bc2d7b3f976d803 (DEFAULT_TABLE table=default.Table_7ba7fd44e98542e78c604739a5086a91 [id=a7a55f9e68f04a9c9c43b9607cd901c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.353356 13274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47b8b57d5b44a018bc2d7b3f976d803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.370574 13691 tablet_bootstrap.cc:492] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:39.371174 13691 ts_tablet_manager.cc:1397] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.986s user 0.026s sys 0.004s
I20260810 23:32:39.375862 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ad5d63a2104995990a86a000963cdc with delete_type TABLET_DATA_DELETED (replaced by tablet 8fb2ee1c9b41478da001aa67d7a7ad3c) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:39.376133 13031 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0136a1e890436fae572b46e64750e7 (DEFAULT_TABLE table=default.Table_32d9165e885d480881ced6b879bcd759 [id=bece3c4a5e8e4dc4bb23129238561321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.377043 13014 tablet_service.cc:1467] Processing CreateTablet for tablet f1e8311c941b43058384cee6d8263dc1 (DEFAULT_TABLE table=default.Table_d8373b61098245bcaab43448a20578f6 [id=7934cc21dac24c74848a668abe1dd606]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.378336 13866 raft_consensus.cc:482] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:39.379086 13866 raft_consensus.cc:504] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:39.384122 13022 tablet_service.cc:1467] Processing CreateTablet for tablet cd24471dd52149d395808493ad9fa68c (DEFAULT_TABLE table=default.Table_241642ff9438427d8938dd88c19ba164 [id=2696163e827d4abca86cb624622710bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.386263 13866 leader_election.cc:288] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:39.385154 14152 tablet.cc:1722] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:39.388489 13007 tablet_service.cc:1467] Processing CreateTablet for tablet 4bce222cd5ab41fdb12fff3fe3dfbd3c (DEFAULT_TABLE table=default.Table_34f40e29ad7a4068943dc4c724bce3b9 [id=3d5ee1f5b27f4707b79ed3fef7b8ffa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.293556 14136 raft_consensus.cc:504] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:39.390153 12235 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:39.391171 14121 tablet_replica.cc:330] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:39.400189 14116 raft_consensus.cc:2726] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:39.405891 14136 leader_election.cc:288] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
W20260810 23:32:39.409958 12952 leader_election.cc:341] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 8b1f7c16b6bc4dc0bac80ac55332f056
I20260810 23:32:39.385782 13691 raft_consensus.cc:348] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:39.416910 12235 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_75ca982c15e34c8d9603f100dff5b392 [id=b809029bf6cc48ab917e48bbf2109ae1]). Expected version 2 got 0
W20260810 23:32:39.421762 13213 leader_election.cc:341] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 1f791b2df0524997ab1c53a9e5cb39ca
I20260810 23:32:39.423965 12767 tablet_service.cc:1467] Processing CreateTablet for tablet f47b8b57d5b44a018bc2d7b3f976d803 (DEFAULT_TABLE table=default.Table_7ba7fd44e98542e78c604739a5086a91 [id=a7a55f9e68f04a9c9c43b9607cd901c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.425632 12767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47b8b57d5b44a018bc2d7b3f976d803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.362733 13974 rpcz_store.cc:276] 0810 23:32:37.720770 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:37.721017 (+ 247us) service_pool.cc:227] Handling call
0810 23:32:39.337517 (+1616500us) ts_tablet_manager.cc:1145] Deleting tablet db363eb0f98d460eb6f5a577233d8ac0
0810 23:32:39.337582 (+ 65us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:39.338048 (+ 466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1596160}
I20260810 23:32:39.411715 13691 raft_consensus.cc:374] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:39.426095 13691 raft_consensus.cc:729] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:39.429833 13023 tablet_service.cc:1467] Processing CreateTablet for tablet f47b8b57d5b44a018bc2d7b3f976d803 (DEFAULT_TABLE table=default.Table_7ba7fd44e98542e78c604739a5086a91 [id=a7a55f9e68f04a9c9c43b9607cd901c6]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:39.416594 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 255147us
I20260810 23:32:39.431471 12958 leader_election.cc:302] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:39.440788 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3049bcdc6e43c795961c74f0d008f5 (DEFAULT_TABLE table=default.Table_7548a45d866747b9b0944e13b211c643 [id=c6af3828fca045bdbe16438171ff83cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.442469 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3049bcdc6e43c795961c74f0d008f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.446573 13514 tablet_bootstrap.cc:492] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:39.447351 13514 ts_tablet_manager.cc:1397] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.468s user 0.011s sys 0.010s
I20260810 23:32:39.452137 12237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet c3d4c879ca6a402dbafe91940e7b367a
I20260810 23:32:39.457592 12768 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d4c879ca6a402dbafe91940e7b367a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:39 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:39.460330 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d4c879ca6a402dbafe91940e7b367a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:39 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:39.463166 14124 raft_consensus.cc:504] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:39.464262 13974 tablet_replica.cc:330] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:39.469889 14124 leader_election.cc:288] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:39.470165 13025 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3049bcdc6e43c795961c74f0d008f5 (DEFAULT_TABLE table=default.Table_7548a45d866747b9b0944e13b211c643 [id=c6af3828fca045bdbe16438171ff83cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.472400 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046f1a7983ec40489edeaba4d2595b1c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:39.474484 14136 raft_consensus.cc:2781] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:39.475438 14136 raft_consensus.cc:686] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:39.476909 14136 consensus_queue.cc:237] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:39.479228 13974 raft_consensus.cc:2229] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:39.469529 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45f9dfd03764d7fb1b6fb5e28b66bd3" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:39.480563 13335 raft_consensus.cc:3037] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:39.440884 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet db363eb0f98d460eb6f5a577233d8ac0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db363eb0f98d460eb6f5a577233d8ac0 already in progress: creating tablet
I20260810 23:32:39.503185 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3049bcdc6e43c795961c74f0d008f5 (DEFAULT_TABLE table=default.Table_7548a45d866747b9b0944e13b211c643 [id=c6af3828fca045bdbe16438171ff83cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.504619 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3049bcdc6e43c795961c74f0d008f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.426860 13691 consensus_queue.cc:260] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:39.507153 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd77526f1db4455291fde8a001092a09" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:39.539695 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:39.541637 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_75ca982c15e34c8d9603f100dff5b392 [id=b809029bf6cc48ab917e48bbf2109ae1]). Expected version 2 got 0
I20260810 23:32:39.542951 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:39.553850 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0136a1e890436fae572b46e64750e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.556790 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e8311c941b43058384cee6d8263dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.558281 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046f1a7983ec40489edeaba4d2595b1c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:39.561408 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd24471dd52149d395808493ad9fa68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.564579 13010 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 92) took 1499 ms. Trace:
I20260810 23:32:39.565588 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d596299c82b4ff1ac151d68b97ff55b" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:39.565207 13010 rpcz_store.cc:276] 0810 23:32:38.064724 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.065626 (+ 902us) service_pool.cc:227] Handling call
0810 23:32:38.094018 (+ 28392us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:38.094421 (+ 403us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:38.791363 (+696942us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:39.350747 (+559384us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:39.563645 (+212898us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:39.567186 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45f9dfd03764d7fb1b6fb5e28b66bd3" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:39.568034 12802 raft_consensus.cc:3037] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:39.570415 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47b8b57d5b44a018bc2d7b3f976d803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.571326 13691 ts_tablet_manager.cc:1428] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.200s user 0.008s sys 0.012s
I20260810 23:32:39.566450 12803 raft_consensus.cc:3037] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:39.493257 12214 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 53296.9us
I20260810 23:32:39.573582 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3049bcdc6e43c795961c74f0d008f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:39.574343 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 0c0d6cd2ad1d4a65929b0d856a171592 because the tablet was not found. No further retry: Not found: Tablet not found: 0c0d6cd2ad1d4a65929b0d856a171592
W20260810 23:32:39.592543 12700 leader_election.cc:341] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:39.595448 12700 leader_election.cc:302] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:39.603072 13018 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 95) took 1370 ms. Trace:
I20260810 23:32:39.603562 13018 rpcz_store.cc:276] 0810 23:32:38.232965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.272119 (+ 39154us) service_pool.cc:227] Handling call
0810 23:32:38.275137 (+ 3018us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:38.275305 (+ 168us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:39.036138 (+760833us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:39.429731 (+393593us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:39.602633 (+172902us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25648}
I20260810 23:32:39.607168 13691 tablet_bootstrap.cc:492] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
W20260810 23:32:39.606196 12700 leader_election.cc:341] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:39.617238 13665 tablet_bootstrap.cc:700] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent opening tablet: real 0.153s user 0.003s sys 0.000s
I20260810 23:32:39.618387 13665 tablet_bootstrap.cc:654] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:39.619221 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 9443fb32d79941d58b13607e8ac064cd (DEFAULT_TABLE table=default.Table_83e4e8027f8b4f61b104e94ccf146481 [id=14bc67f80e344860827f60af309df4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.620414 13974 raft_consensus.cc:2258] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:39.624338 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9443fb32d79941d58b13607e8ac064cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.629230 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_75ca982c15e34c8d9603f100dff5b392 [id=b809029bf6cc48ab917e48bbf2109ae1]). Expected version 2 got 0
I20260810 23:32:39.629815 13007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bce222cd5ab41fdb12fff3fe3dfbd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.648106 14121 raft_consensus.cc:2229] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:39.648907 14121 raft_consensus.cc:2258] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:39.655737 14124 raft_consensus.cc:482] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:39.656385 14124 raft_consensus.cc:504] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:39.681998 14145 raft_consensus.cc:482] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:39.682621 14145 raft_consensus.cc:504] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:39.684082 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 67379.1us
W20260810 23:32:39.686120 12700 leader_election.cc:341] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:39.672610 14121 ts_tablet_manager.cc:1905] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:39.690145 13514 raft_consensus.cc:348] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:39.691102 13514 raft_consensus.cc:374] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:39.691365 13514 raft_consensus.cc:729] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:39.693449 13514 consensus_queue.cc:260] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:39.698477 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 9443fb32d79941d58b13607e8ac064cd (DEFAULT_TABLE table=default.Table_83e4e8027f8b4f61b104e94ccf146481 [id=14bc67f80e344860827f60af309df4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:39.682914 14107 raft_consensus.cc:2726] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:39.699636 13213 consensus_peers.cc:487] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81e26fe4d42445279f12bb17b2841d08. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:39.699839 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9443fb32d79941d58b13607e8ac064cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.712014 13514 ts_tablet_manager.cc:1428] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.264s user 0.005s sys 0.010s
I20260810 23:32:39.713964 14101 raft_consensus.cc:482] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:39.714694 14101 raft_consensus.cc:504] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:39.726233 14101 leader_election.cc:288] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:39.727417 13514 tablet_bootstrap.cc:492] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:39.737717 14152 tablet.cc:1717] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3: Skipping requested alter to schema version 2, tablet already version 2
I20260810 23:32:39.709594 14124 leader_election.cc:288] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:39.759212 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081b4e1f53c74f768a4bf401ae22b585" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:39.762434 12952 leader_election.cc:341] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 0c0d6cd2ad1d4a65929b0d856a171592
I20260810 23:32:39.776537 13935 ts_tablet_manager.cc:1918] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:39.760633 14153 tablet.cc:1722] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:39.779197 13935 log.cc:1196] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/840400119d20424ea0f03f2bff550d44
I20260810 23:32:39.781019 14136 raft_consensus.cc:482] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:39.781780 14136 raft_consensus.cc:504] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:39.782883 13935 ts_tablet_manager.cc:1939] T 840400119d20424ea0f03f2bff550d44 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:39.751551 14145 leader_election.cc:288] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:39.784612 14136 leader_election.cc:288] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
W20260810 23:32:39.803387 12698 leader_election.cc:341] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:39.805042 14101 raft_consensus.cc:482] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:39.805903 14101 raft_consensus.cc:504] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:39.806763 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046f1a7983ec40489edeaba4d2595b1c" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:39.813627 14101 leader_election.cc:288] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
W20260810 23:32:39.818477 12952 leader_election.cc:341] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 79da9583d3524347a9be09e35affaaad
I20260810 23:32:39.820910 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874ca0c3586b41a8a3074d8988813633" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:39.823895 12952 leader_election.cc:341] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:39.826905 12700 leader_election.cc:341] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:39.827535 12700 leader_election.cc:302] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:39.830824 14101 raft_consensus.cc:2726] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:39.833671 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f935119e4d9b4accb79db98185c80aca" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:39.845224 12698 leader_election.cc:341] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:39.851627 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f791b2df0524997ab1c53a9e5cb39ca" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:39.853084 13060 raft_consensus.cc:2454] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:39.861552 13514 tablet_bootstrap.cc:654] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:39.876272 13691 tablet_bootstrap.cc:654] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:39.889022 13974 ts_tablet_manager.cc:1905] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:39.908746 12802 raft_consensus.cc:2454] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:39.910470 12803 raft_consensus.cc:2454] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:39.916590 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 82) took 3563 ms. Trace:
I20260810 23:32:39.917224 13935 rpcz_store.cc:276] 0810 23:32:36.353411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.353823 (+ 412us) service_pool.cc:227] Handling call
0810 23:32:38.954171 (+2600348us) ts_tablet_manager.cc:1145] Deleting tablet 840400119d20424ea0f03f2bff550d44
0810 23:32:38.954223 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:39.187293 (+233070us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:39.776219 (+588926us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:39.916501 (+140282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2599064}
I20260810 23:32:39.918761 13935 tablet_replica.cc:330] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:39.919579 13935 raft_consensus.cc:2229] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:39.928966 13935 raft_consensus.cc:2258] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:39.934262 13935 ts_tablet_manager.cc:1905] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:39.957840 12698 consensus_peers.cc:487] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:39.987478 13212 leader_election.cc:341] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:39.988065 13212 leader_election.cc:302] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:39.988117 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081b4e1f53c74f768a4bf401ae22b585" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:39.989439 14041 raft_consensus.cc:2726] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:39.990257 14041 consensus_queue.cc:1035] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:40.064491 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f935119e4d9b4accb79db98185c80aca" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:40.083504 13335 consensus_meta.cc:287] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent flushing consensus metadata: real 0.602s user 0.008s sys 0.003s
W20260810 23:32:39.742115 12214 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 166115us
W20260810 23:32:39.882887 12218 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 4d697e7533af4fc0a15cb41f3fcce4db (table default.Table_26bf2c7d86704b5b981ecebb4e90ba75 [id=cefe7b97c1db42ef8712bf3c7a359af2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4d697e7533af4fc0a15cb41f3fcce4db consensus state has no leader
I20260810 23:32:39.925410 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 840400119d20424ea0f03f2bff550d44 (table default.Table_e0603d4692934dce9e07597811ccad0a [id=ad699de357934075aee8d7152f8e130b]) successfully deleted
I20260810 23:32:40.085304 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
W20260810 23:32:40.085768 12217 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 160028us
W20260810 23:32:40.100514 12224 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a3a834d77474c3d8feaa5a01f3faff3 consensus state has no leader
I20260810 23:32:40.101181 12224 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 180) took 1523 ms. Trace:
I20260810 23:32:40.101568 12224 rpcz_store.cc:276] 0810 23:32:38.577465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:39.303714 (+726249us) service_pool.cc:227] Handling call
0810 23:32:39.330162 (+ 26448us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:39.331714 (+ 1552us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:39.332582 (+ 868us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:39.332914 (+ 332us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:40.099605 (+766691us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:40.101082 (+ 1477us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:39.713182 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:39.743700 (+ 30518us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:39.743772 (+ 72us) write_op.cc:435] Acquired schema lock
0810 23:32:39.743807 (+ 35us) tablet.cc:660] Decoding operations
0810 23:32:39.744630 (+ 823us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:39.744750 (+ 120us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:39.744808 (+ 58us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:39.745190 (+ 382us) tablet.cc:699] Row locks acquired
0810 23:32:39.745213 (+ 23us) write_op.cc:260] PREPARE: finished
0810 23:32:39.827248 (+ 82035us) write_op.cc:270] Start()
0810 23:32:39.827707 (+ 459us) write_op.cc:276] Timestamp: P: 1786404759745379 usec, L: 0
0810 23:32:39.827733 (+ 26us) op_driver.cc:331] REPLICATION: starting
0810 23:32:39.869780 (+ 42047us) log.cc:842] Serialized 405 byte log entry
0810 23:32:40.096643 (+226863us) op_driver.cc:444] REPLICATION: finished
0810 23:32:40.097170 (+ 527us) write_op.cc:301] APPLY: starting
0810 23:32:40.097293 (+ 123us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:40.097480 (+ 187us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:40.097670 (+ 190us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:40.097950 (+ 280us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:40.098215 (+ 265us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:40.098306 (+ 91us) write_op.cc:312] APPLY: finished
0810 23:32:40.098576 (+ 270us) log.cc:842] Serialized 23 byte log entry
0810 23:32:40.098819 (+ 243us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:40.098918 (+ 99us) write_op.cc:454] Released schema lock
0810 23:32:40.098984 (+ 66us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:40.099308 (+ 324us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:40.099364 (+ 56us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":252,"num_ops":1,"prepare.queue_time_us":378473,"prepare.run_cpu_time_us":3717,"prepare.run_wall_time_us":157119,"raft.queue_time_us":3581,"raft.run_cpu_time_us":1024,"raft.run_wall_time_us":1025,"replication_time_us":268781}]]}
I20260810 23:32:40.095736 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874ca0c3586b41a8a3074d8988813633" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:40.117710 12806 raft_consensus.cc:2454] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
W20260810 23:32:40.121176 12767 consensus_meta.cc:287] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.526s user 0.003s sys 0.011s
I20260810 23:32:40.129053 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_96019e28037b4d38aa9fc933a29c8884"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:40.131350 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96019e28037b4d38aa9fc933a29c8884 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:40.047463 12958 leader_election.cc:341] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 8b1f7c16b6bc4dc0bac80ac55332f056
I20260810 23:32:40.154436 12958 leader_election.cc:302] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:32:40.155572 12958 raft_consensus.cc:2670] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260810 23:32:40.156322 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 109984us
I20260810 23:32:40.158165 12958 leader_election.cc:302] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:40.159047 14136 raft_consensus.cc:2781] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:40.159576 14136 raft_consensus.cc:686] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:40.160742 14136 consensus_queue.cc:237] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:40.099751 13974 ts_tablet_manager.cc:1918] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:40.123026 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d4c879ca6a402dbafe91940e7b367a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:39 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:40.125682 13274 tablet_service.cc:1467] Processing CreateTablet for tablet 9443fb32d79941d58b13607e8ac064cd (DEFAULT_TABLE table=default.Table_83e4e8027f8b4f61b104e94ccf146481 [id=14bc67f80e344860827f60af309df4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.260528 13974 log.cc:1196] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/dd83bbe7a1664c278098fa66b9c6895e
I20260810 23:32:40.262722 13974 ts_tablet_manager.cc:1939] T dd83bbe7a1664c278098fa66b9c6895e P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:40.265151 13274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9443fb32d79941d58b13607e8ac064cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.188277 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 874ca0c3586b41a8a3074d8988813633 (table default.Table_b57f0f6e94384d22af5d3bf7817d5044 [id=4bb9a6fbe6e644c583a468af86950cef]). Expected version 1 got 0
I20260810 23:32:40.266465 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:40.267549 12236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 9c29b11192604ac18c21b7bdc0238fb3
I20260810 23:32:40.192588 12224 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 181) took 1285 ms. Trace:
I20260810 23:32:40.268424 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 107) took 1597 ms. Trace:
I20260810 23:32:40.269384 12236 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 178) took 1769 ms. Trace:
I20260810 23:32:40.268934 13974 rpcz_store.cc:276] 0810 23:32:38.667057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.669103 (+ 2046us) service_pool.cc:227] Handling call
0810 23:32:39.427789 (+758686us) ts_tablet_manager.cc:1145] Deleting tablet dd83bbe7a1664c278098fa66b9c6895e
0810 23:32:39.427842 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:40.058508 (+630666us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:40.083834 (+ 25326us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:40.263985 (+180151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":757825}
I20260810 23:32:40.271517 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 110) took 1602 ms. Trace:
I20260810 23:32:40.274827 12228 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_94e24b4b27d047a99df427c453fe1752 [id=e462d54230344223a457f708501d2d30] alter complete (version 3)
I20260810 23:32:40.276163 13974 rpcz_store.cc:276] 0810 23:32:38.669385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.669499 (+ 114us) service_pool.cc:227] Handling call
0810 23:32:40.270978 (+1601479us) ts_tablet_manager.cc:1145] Deleting tablet 79da9583d3524347a9be09e35affaaad
0810 23:32:40.271097 (+ 119us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:40.271442 (+ 345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600804}
I20260810 23:32:40.277834 13974 tablet_replica.cc:330] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:40.278777 13974 raft_consensus.cc:2229] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:40.279076 13514 tablet_bootstrap.cc:492] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:40.279740 13974 raft_consensus.cc:2258] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:40.279925 13514 ts_tablet_manager.cc:1397] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.553s user 0.021s sys 0.004s
I20260810 23:32:40.294719 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c29b11192604ac18c21b7bdc0238fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:38 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:40.296545 12767 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c29b11192604ac18c21b7bdc0238fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:38 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:40.269698 12236 rpcz_store.cc:276] 0810 23:32:38.500480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.923246 (+422766us) service_pool.cc:227] Handling call
0810 23:32:38.963814 (+ 40568us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:38.964584 (+ 770us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:38.964835 (+ 251us) catalog_manager.cc:2761] Locking tablets
0810 23:32:39.234102 (+269267us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:39.655623 (+421521us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:39.800379 (+144756us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:40.267406 (+467027us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:40.269310 (+ 1904us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:39.397865 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:39.460043 (+ 62178us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:39.460089 (+ 46us) write_op.cc:435] Acquired schema lock
0810 23:32:39.460113 (+ 24us) tablet.cc:660] Decoding operations
0810 23:32:39.460721 (+ 608us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:39.460790 (+ 69us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:39.460829 (+ 39us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:39.498727 (+ 37898us) tablet.cc:699] Row locks acquired
0810 23:32:39.498753 (+ 26us) write_op.cc:260] PREPARE: finished
0810 23:32:39.498953 (+ 200us) write_op.cc:270] Start()
0810 23:32:39.499080 (+ 127us) write_op.cc:276] Timestamp: P: 1786404759498854 usec, L: 0
0810 23:32:39.499105 (+ 25us) op_driver.cc:331] REPLICATION: starting
0810 23:32:39.499965 (+ 860us) log.cc:842] Serialized 765 byte log entry
0810 23:32:39.650948 (+150983us) op_driver.cc:444] REPLICATION: finished
0810 23:32:39.651585 (+ 637us) write_op.cc:301] APPLY: starting
0810 23:32:39.651689 (+ 104us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:39.651925 (+ 236us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:39.652114 (+ 189us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:39.652596 (+ 482us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:39.652927 (+ 331us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:39.653018 (+ 91us) write_op.cc:312] APPLY: finished
0810 23:32:39.653389 (+ 371us) log.cc:842] Serialized 29 byte log entry
0810 23:32:39.653631 (+ 242us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:39.653749 (+ 118us) write_op.cc:454] Released schema lock
0810 23:32:39.653816 (+ 67us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:39.654173 (+ 357us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:39.654221 (+ 48us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":330,"apply.run_cpu_time_us":2958,"apply.run_wall_time_us":2968,"num_ops":2,"prepare.queue_time_us":141168,"prepare.run_cpu_time_us":4161,"prepare.run_wall_time_us":315127,"raft.queue_time_us":136019,"raft.run_cpu_time_us":1428,"raft.run_wall_time_us":1429,"replication_time_us":151728}]]}
I20260810 23:32:40.298468 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c29b11192604ac18c21b7bdc0238fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:38 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:40.285006 13335 raft_consensus.cc:2454] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:40.302018 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_d4fad7cecea149e88811df34c8c11d2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:40.305387 12770 consensus_meta.cc:287] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.866s user 0.013s sys 0.000s
W20260810 23:32:40.306041 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4fad7cecea149e88811df34c8c11d2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:40.315442 14153 tablet.cc:1717] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3: Skipping requested alter to schema version 2, tablet already version 2
I20260810 23:32:40.321300 13974 ts_tablet_manager.cc:1905] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:40.268342 12224 rpcz_store.cc:276] 0810 23:32:38.907412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.119485 (+1212073us) service_pool.cc:227] Handling call
0810 23:32:40.131989 (+ 12504us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:40.137752 (+ 5763us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:40.189882 (+ 52130us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:40.190128 (+ 246us) write_op.cc:454] Released schema lock
0810 23:32:40.190476 (+ 348us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:40.191192 (+ 716us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:40.192486 (+ 1294us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:40.143639 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:40.144301 (+ 662us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:40.144329 (+ 28us) write_op.cc:435] Acquired schema lock
0810 23:32:40.144345 (+ 16us) tablet.cc:660] Decoding operations
0810 23:32:40.144698 (+ 353us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:40.144751 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:40.144777 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:40.145021 (+ 244us) tablet.cc:699] Row locks acquired
0810 23:32:40.145035 (+ 14us) write_op.cc:260] PREPARE: finished
0810 23:32:40.145172 (+ 137us) write_op.cc:270] Start()
0810 23:32:40.145279 (+ 107us) write_op.cc:276] Timestamp: P: 1786404760145098 usec, L: 0
0810 23:32:40.145296 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:32:40.145928 (+ 632us) log.cc:842] Serialized 433 byte log entry
0810 23:32:40.163064 (+ 17136us) op_driver.cc:444] REPLICATION: finished
0810 23:32:40.165158 (+ 2094us) write_op.cc:301] APPLY: starting
0810 23:32:40.165244 (+ 86us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:40.165416 (+ 172us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:40.165554 (+ 138us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:40.166166 (+ 612us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:40.166371 (+ 205us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:40.166443 (+ 72us) write_op.cc:312] APPLY: finished
0810 23:32:40.166727 (+ 284us) log.cc:842] Serialized 29 byte log entry
0810 23:32:40.166931 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:40.167023 (+ 92us) write_op.cc:454] Released schema lock
0810 23:32:40.167064 (+ 41us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1830,"apply.run_cpu_time_us":2197,"apply.run_wall_time_us":2196,"num_ops":2,"prepare.queue_time_us":842,"prepare.run_cpu_time_us":2506,"prepare.run_wall_time_us":2667,"raft.queue_time_us":823,"raft.run_cpu_time_us":1092,"raft.run_wall_time_us":1092,"replication_time_us":17642}]]}
I20260810 23:32:40.328930 12224 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_3e86c4915aac433bb544495b6bd22c1c" } new_table_name: "default.Table_dc5d83ea7f1441fa9e2b2536bab49724" modify_external_catalogs: true
W20260810 23:32:40.333841 12952 consensus_peers.cc:487] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:40.335178 13691 tablet_bootstrap.cc:492] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:40.335767 13691 ts_tablet_manager.cc:1397] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.729s user 0.008s sys 0.017s
I20260810 23:32:40.247784 12958 leader_election.cc:302] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:32:40.348924 13057 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.028s sys 0.068s
I20260810 23:32:40.331218 13665 tablet_bootstrap.cc:492] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:40.364427 13665 ts_tablet_manager.cc:1397] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 1.152s user 0.019s sys 0.003s
W20260810 23:32:40.370713 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 159874us
W20260810 23:32:40.372175 12958 leader_election.cc:341] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 79da9583d3524347a9be09e35affaaad
I20260810 23:32:40.372937 12958 leader_election.cc:302] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:40.385506 13290 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 118) took 1083 ms. Trace:
W20260810 23:32:40.413241 12223 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 40) took 5016 ms (client timeout 1000 ms). Trace:
I20260810 23:32:40.414410 13290 rpcz_store.cc:276] 0810 23:32:39.302184 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:39.302387 (+ 203us) service_pool.cc:227] Handling call
0810 23:32:39.325103 (+ 22716us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:39.329037 (+ 3934us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:40.075352 (+746315us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:40.384854 (+309502us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:40.385419 (+ 565us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:40.419107 14145 raft_consensus.cc:2726] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:32:40.413199 13665 raft_consensus.cc:348] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:40.420163 13665 raft_consensus.cc:374] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:40.420013 14136 raft_consensus.cc:2781] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:40.420603 13665 raft_consensus.cc:729] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:40.420987 14136 raft_consensus.cc:686] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:40.421708 13665 consensus_queue.cc:260] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:40.422315 14136 consensus_queue.cc:237] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:40.428643 13935 ts_tablet_manager.cc:1918] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:40.429227 13935 log.cc:1196] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/3d596299c82b4ff1ac151d68b97ff55b
I20260810 23:32:40.430332 13935 ts_tablet_manager.cc:1939] T 3d596299c82b4ff1ac151d68b97ff55b P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
W20260810 23:32:40.374538 13057 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.579s user 0.028s sys 0.068s
I20260810 23:32:40.446514 13665 ts_tablet_manager.cc:1428] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.082s user 0.007s sys 0.002s
I20260810 23:32:40.449463 13665 tablet_bootstrap.cc:492] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:40.439320 13273 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 115) took 1196 ms. Trace:
I20260810 23:32:40.464076 13273 rpcz_store.cc:276] 0810 23:32:39.243371 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:39.257293 (+ 13922us) service_pool.cc:227] Handling call
0810 23:32:39.259404 (+ 2111us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:39.259790 (+ 386us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:40.168280 (+908490us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:40.438589 (+270309us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:40.439247 (+ 658us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:40.506207 13691 raft_consensus.cc:348] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:40.507174 13691 raft_consensus.cc:374] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:40.510342 13665 tablet_bootstrap.cc:654] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:40.507519 13691 raft_consensus.cc:729] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:40.511358 13691 consensus_queue.cc:260] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:40.521786 12770 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 113) took 1211 ms. Trace:
I20260810 23:32:40.525802 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 87) took 4004 ms. Trace:
I20260810 23:32:40.526221 13935 rpcz_store.cc:276] 0810 23:32:36.521266 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.522505 (+ 1239us) service_pool.cc:227] Handling call
0810 23:32:39.918579 (+3396074us) ts_tablet_manager.cc:1145] Deleting tablet 3d596299c82b4ff1ac151d68b97ff55b
0810 23:32:39.918625 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:40.241483 (+322858us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:40.428159 (+186676us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:40.525720 (+ 97561us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13568,"tablet-delete.queue_time_us":3394634}
I20260810 23:32:40.527901 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 91) took 3888 ms. Trace:
I20260810 23:32:40.528589 13935 rpcz_store.cc:276] 0810 23:32:36.640090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.640510 (+ 420us) service_pool.cc:227] Handling call
0810 23:32:40.527453 (+3886943us) ts_tablet_manager.cc:1145] Deleting tablet b261673faa964b65af3576c89e302864
0810 23:32:40.527519 (+ 66us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:40.527838 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3884754}
I20260810 23:32:40.529946 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 93) took 3371 ms. Trace:
I20260810 23:32:40.522163 12770 rpcz_store.cc:276] 0810 23:32:39.310676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:39.310933 (+ 257us) service_pool.cc:227] Handling call
0810 23:32:39.312622 (+ 1689us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:39.312714 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:39.438655 (+125941us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:40.306912 (+868257us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:40.521699 (+214787us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:40.530238 13935 rpcz_store.cc:276] 0810 23:32:37.158324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:37.158648 (+ 324us) service_pool.cc:227] Handling call
0810 23:32:40.529596 (+3370948us) ts_tablet_manager.cc:1145] Deleting tablet db363eb0f98d460eb6f5a577233d8ac0
0810 23:32:40.529636 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:40.529891 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3369681}
I20260810 23:32:40.531193 13935 tablet_replica.cc:330] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:40.534163 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f791b2df0524997ab1c53a9e5cb39ca with delete_type TABLET_DATA_DELETED (replaced by tablet c892e6bc117a4fff917342b8eefaf74d) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:40.531958 13935 raft_consensus.cc:2229] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:40.540282 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet fd110664bac34f52bfe922a3a76c6290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:40.541605 13014 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e26fe4d42445279f12bb17b2841d08 with delete_type TABLET_DATA_DELETED (replaced by tablet b261673faa964b65af3576c89e302864) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:40.542665 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet 79da9583d3524347a9be09e35affaaad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:40.438016 12236 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 182) took 1442 ms. Trace:
I20260810 23:32:40.558933 13514 raft_consensus.cc:348] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:40.559789 13514 raft_consensus.cc:374] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 23:32:40.441321 12224 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd9d90e1ceab443dbec12b2c81b975e8 consensus state has no leader
I20260810 23:32:40.560072 13514 raft_consensus.cc:729] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:40.560453 12224 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 183) took 1489 ms. Trace:
I20260810 23:32:40.575173 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b (DEFAULT_TABLE table=default.Table_d942bf134f9849718bb9eb12c24806a1 [id=ff8088bdfc9f456683dd802d62ae67a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.577386 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.292901 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet dd83bbe7a1664c278098fa66b9c6895e (table default.Table_7c223ce60be04a498d08155325785819 [id=0eeab77fa3d54892a91f87f2fbdd673d]) successfully deleted
W20260810 23:32:40.580415 12214 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 287546us
W20260810 23:32:40.581880 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 79da9583d3524347a9be09e35affaaad because the tablet was not found. No further retry: Not found: Tablet not found: 79da9583d3524347a9be09e35affaaad
I20260810 23:32:40.581982 13014 tablet_service.cc:1467] Processing CreateTablet for tablet 240adfa4be844518bbc9624d0e408789 (DEFAULT_TABLE table=default.Table_32d7e9316d054deca361c6e3d9b5e1f4 [id=43ef82e91d4a4e2b8bb06d4cab61c8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.584178 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240adfa4be844518bbc9624d0e408789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.591631 13025 tablet_service.cc:1467] Processing CreateTablet for tablet a41ddf9f6f71443ba75c97d2b010372c (DEFAULT_TABLE table=default.Table_73f2b1b207fb45f2adecbd37412c5e6f [id=c57f7c273a9841c1ac02523a733a383f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.593994 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41ddf9f6f71443ba75c97d2b010372c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.593765 13031 tablet_service.cc:1467] Processing CreateTablet for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (DEFAULT_TABLE table=default.Table_34eb8fd429274b5d8af145c70aab107b [id=dae244793fd24816a8199c38f23a25b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.596252 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.596145 13023 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (DEFAULT_TABLE table=default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.596654 13022 tablet_service.cc:1467] Processing CreateTablet for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (DEFAULT_TABLE table=default.Table_d6f432a771c544d6acfb28ec150340da [id=2c8320255aab4036b5350d082f9e04ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.617092 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7bd9b6ca4845d8b8705ff970c8f06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.618228 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1b9d20d09c4d1ba9a4b10959807e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.619402 13935 raft_consensus.cc:2258] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:40.626730 13935 ts_tablet_manager.cc:1905] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:40.636644 13514 consensus_queue.cc:260] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:40.647428 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0d6cd2ad1d4a65929b0d856a171592 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:40.650534 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e26fe4d42445279f12bb17b2841d08 with delete_type TABLET_DATA_DELETED (replaced by tablet b261673faa964b65af3576c89e302864) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:40.652812 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (DEFAULT_TABLE table=default.Table_34eb8fd429274b5d8af145c70aab107b [id=dae244793fd24816a8199c38f23a25b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.654176 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.657123 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet b8986e59b2314bd69b9b1de36f1f2ba7 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d697e7533af4fc0a15cb41f3fcce4db) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:40.659227 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 240adfa4be844518bbc9624d0e408789 (DEFAULT_TABLE table=default.Table_32d7e9316d054deca361c6e3d9b5e1f4 [id=43ef82e91d4a4e2b8bb06d4cab61c8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.661304 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240adfa4be844518bbc9624d0e408789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.664719 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:40.665876 13273 tablet_service.cc:1514] Processing DeleteTablet for tablet fd110664bac34f52bfe922a3a76c6290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:40.505330 12229 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 41) took 3842 ms (client timeout 1000 ms). Trace:
W20260810 23:32:40.539042 12228 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 26) took 3093 ms (client timeout 1000 ms). Trace:
W20260810 23:32:40.677989 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 305822us
W20260810 23:32:40.679361 12958 leader_election.cc:341] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 0c0d6cd2ad1d4a65929b0d856a171592
W20260810 23:32:40.541183 12226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 19) took 1475 ms (client timeout 1000 ms). Trace:
W20260810 23:32:40.680161 12226 rpcz_store.cc:269] 0810 23:32:39.065894 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:39.113070 (+ 47176us) service_pool.cc:227] Handling call
0810 23:32:40.540936 (+1427866us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":119761,"reported_tablets":12}
W20260810 23:32:40.542639 12234 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 25) took 4093 ms (client timeout 1000 ms). Trace:
W20260810 23:32:40.685561 12234 rpcz_store.cc:269] 0810 23:32:36.449264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.607644 (+158380us) service_pool.cc:227] Handling call
0810 23:32:40.542547 (+3934903us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":148400,"reported_tablets":10}
W20260810 23:32:40.543690 12240 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 18) took 3345 ms (client timeout 1000 ms). Trace:
W20260810 23:32:40.500804 12235 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 42) took 2246 ms (client timeout 1000 ms). Trace:
W20260810 23:32:40.688704 12235 rpcz_store.cc:269] 0810 23:32:38.254833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.297060 (+ 42227us) service_pool.cc:227] Handling call
0810 23:32:40.500664 (+2203604us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35368,"reported_tablets":16}
I20260810 23:32:40.557801 12236 rpcz_store.cc:276] 0810 23:32:38.996111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.301224 (+1305113us) service_pool.cc:227] Handling call
0810 23:32:40.310139 (+ 8915us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:40.385814 (+ 75675us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:40.437160 (+ 51346us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:40.437596 (+ 436us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:40.437927 (+ 331us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:40.387603 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:40.387993 (+ 390us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:40.388041 (+ 48us) write_op.cc:435] Acquired schema lock
0810 23:32:40.388053 (+ 12us) tablet.cc:660] Decoding operations
0810 23:32:40.388328 (+ 275us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:40.388382 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:40.388403 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:40.388613 (+ 210us) tablet.cc:699] Row locks acquired
0810 23:32:40.388627 (+ 14us) write_op.cc:260] PREPARE: finished
0810 23:32:40.388745 (+ 118us) write_op.cc:270] Start()
0810 23:32:40.388831 (+ 86us) write_op.cc:276] Timestamp: P: 1786404760388659 usec, L: 0
0810 23:32:40.388848 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:32:40.389477 (+ 629us) log.cc:842] Serialized 433 byte log entry
0810 23:32:40.431500 (+ 42023us) op_driver.cc:444] REPLICATION: finished
0810 23:32:40.432008 (+ 508us) write_op.cc:301] APPLY: starting
0810 23:32:40.432133 (+ 125us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:40.432342 (+ 209us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:40.432694 (+ 352us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:40.433483 (+ 789us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:40.433689 (+ 206us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:40.433780 (+ 91us) write_op.cc:312] APPLY: finished
0810 23:32:40.434107 (+ 327us) log.cc:842] Serialized 29 byte log entry
0810 23:32:40.434358 (+ 251us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:40.434455 (+ 97us) write_op.cc:454] Released schema lock
0810 23:32:40.434520 (+ 65us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:40.434858 (+ 338us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:40.434912 (+ 54us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":192,"apply.run_cpu_time_us":3172,"apply.run_wall_time_us":3171,"num_ops":2,"prepare.queue_time_us":120,"prepare.run_cpu_time_us":2159,"prepare.run_wall_time_us":2161,"raft.queue_time_us":1466,"raft.run_cpu_time_us":1154,"raft.run_wall_time_us":1155,"replication_time_us":42477}]]}
I20260810 23:32:40.560840 12224 rpcz_store.cc:276] 0810 23:32:39.071603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.328513 (+1256910us) service_pool.cc:227] Handling call
0810 23:32:40.329813 (+ 1300us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:40.393772 (+ 63959us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:40.394364 (+ 592us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:40.394591 (+ 227us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:40.440563 (+ 45972us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:40.560375 (+119812us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:40.397864 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:40.398244 (+ 380us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:40.398268 (+ 24us) write_op.cc:435] Acquired schema lock
0810 23:32:40.398279 (+ 11us) tablet.cc:660] Decoding operations
0810 23:32:40.398538 (+ 259us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:40.398575 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:40.398596 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:40.398737 (+ 141us) tablet.cc:699] Row locks acquired
0810 23:32:40.398751 (+ 14us) write_op.cc:260] PREPARE: finished
0810 23:32:40.398868 (+ 117us) write_op.cc:270] Start()
0810 23:32:40.398965 (+ 97us) write_op.cc:276] Timestamp: P: 1786404760398777 usec, L: 0
0810 23:32:40.398980 (+ 15us) op_driver.cc:331] REPLICATION: starting
0810 23:32:40.399585 (+ 605us) log.cc:842] Serialized 405 byte log entry
0810 23:32:40.436922 (+ 37337us) op_driver.cc:444] REPLICATION: finished
0810 23:32:40.437790 (+ 868us) write_op.cc:301] APPLY: starting
0810 23:32:40.437844 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:40.437976 (+ 132us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:40.438144 (+ 168us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:40.438351 (+ 207us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:40.438631 (+ 280us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:40.438673 (+ 42us) write_op.cc:312] APPLY: finished
0810 23:32:40.438951 (+ 278us) log.cc:842] Serialized 23 byte log entry
0810 23:32:40.439481 (+ 530us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:40.439611 (+ 130us) write_op.cc:454] Released schema lock
0810 23:32:40.439657 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:40.501412 (+ 61755us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:40.501489 (+ 77us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":637,"apply.run_cpu_time_us":2336,"apply.run_wall_time_us":64020,"num_ops":1,"prepare.queue_time_us":2018,"prepare.run_cpu_time_us":2065,"prepare.run_wall_time_us":32657,"raft.queue_time_us":185,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":378,"replication_time_us":37834}]]}
W20260810 23:32:40.655953 12237 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 43) took 1315 ms (client timeout 1000 ms). Trace:
I20260810 23:32:40.691560 12224 master_service.cc:946] ReplaceTablet: received request to replace tablet 42517f11422b42c39db282e2c5f584a3 from {username='slave'} at 127.0.0.1:34742
W20260810 23:32:40.691272 12237 rpcz_store.cc:269] 0810 23:32:39.340454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:39.455299 (+114845us) service_pool.cc:227] Handling call
0810 23:32:40.655836 (+1200537us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11341,"reported_tablets":18}
I20260810 23:32:40.698257 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ba75f2c2b54ec1968db8feeacf8383 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c29b11192604ac18c21b7bdc0238fb3) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:40.700132 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b08ed744b3420f98d241ad96bca6d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d0d9227d8574de8bd73041f3e8043a0) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:40.700980 12237 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_9b23c2d0273b4a1193b49dbf9b79f985" } new_table_name: "default.Table_7ed38c2d149c496393381b8394b2ee06" modify_external_catalogs: true
I20260810 23:32:40.702338 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f791b2df0524997ab1c53a9e5cb39ca with delete_type TABLET_DATA_DELETED (replaced by tablet c892e6bc117a4fff917342b8eefaf74d) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:40.703949 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 840400119d20424ea0f03f2bff550d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:40.712083 13007 tablet_service.cc:1467] Processing CreateTablet for tablet 62705f99687340aaa1aca0942b28487f (DEFAULT_TABLE table=default.Table_96019e28037b4d38aa9fc933a29c8884 [id=dd7ca129aac0430ba386c53a30052148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.719137 13514 ts_tablet_manager.cc:1428] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.439s user 0.007s sys 0.000s
I20260810 23:32:40.712693 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 684caf5aaa8347e8b4e02323318fb0b7 (DEFAULT_TABLE table=default.Table_d4fad7cecea149e88811df34c8c11d2d [id=997d599555454bbca33e0b1d98f2f88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.721850 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 684caf5aaa8347e8b4e02323318fb0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.724248 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_78e8fac0b43b408abf61639343c2e8e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:40.727046 12236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_994c5ce0643d422595ad55d7f893ae9d" } modify_external_catalogs: true
I20260810 23:32:40.727712 12236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_994c5ce0643d422595ad55d7f893ae9d" } modify_external_catalogs: true
I20260810 23:32:40.729688 14178 consensus_queue.cc:1035] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:40.740325 13007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62705f99687340aaa1aca0942b28487f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:40.749042 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78e8fac0b43b408abf61639343c2e8e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:40.755402 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_20b50818a91248faac361d538df4fb30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:40.757107 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20b50818a91248faac361d538df4fb30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:40.758913 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet fd110664bac34f52bfe922a3a76c6290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:40.760207 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e26fe4d42445279f12bb17b2841d08 with delete_type TABLET_DATA_DELETED (replaced by tablet b261673faa964b65af3576c89e302864) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:40.762243 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 79da9583d3524347a9be09e35affaaad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:40.770797 13665 tablet_bootstrap.cc:492] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:40.771304 13665 ts_tablet_manager.cc:1397] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.322s user 0.006s sys 0.013s
I20260810 23:32:40.777038 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0d6cd2ad1d4a65929b0d856a171592 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:40.779043 13514 tablet_bootstrap.cc:492] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:40.780629 12224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 42517f11422b42c39db282e2c5f584a3
W20260810 23:32:40.692898 12240 rpcz_store.cc:269] 0810 23:32:37.198073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:37.278011 (+ 79938us) service_pool.cc:227] Handling call
0810 23:32:40.543608 (+3265597us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:38.060259 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:38.060949 (+ 690us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:38.060982 (+ 33us) write_op.cc:435] Acquired schema lock
0810 23:32:38.060992 (+ 10us) tablet.cc:660] Decoding operations
0810 23:32:38.061368 (+ 376us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:38.061514 (+ 146us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:38.061553 (+ 39us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:38.062055 (+ 502us) tablet.cc:699] Row locks acquired
0810 23:32:38.062076 (+ 21us) write_op.cc:260] PREPARE: finished
0810 23:32:38.062265 (+ 189us) write_op.cc:270] Start()
0810 23:32:38.062570 (+ 305us) write_op.cc:276] Timestamp: P: 1786404758062207 usec, L: 0
0810 23:32:38.062609 (+ 39us) op_driver.cc:331] REPLICATION: starting
0810 23:32:38.063530 (+ 921us) log.cc:842] Serialized 919 byte log entry
0810 23:32:38.155869 (+ 92339us) op_driver.cc:444] REPLICATION: finished
0810 23:32:38.157202 (+ 1333us) write_op.cc:301] APPLY: starting
0810 23:32:38.157335 (+ 133us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:38.157610 (+ 275us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:38.157815 (+ 205us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:38.158358 (+ 543us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:38.158628 (+ 270us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:38.158734 (+ 106us) write_op.cc:312] APPLY: finished
0810 23:32:38.170086 (+ 11352us) log.cc:842] Serialized 29 byte log entry
0810 23:32:38.170400 (+ 314us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:38.170558 (+ 158us) write_op.cc:454] Released schema lock
0810 23:32:38.170724 (+ 166us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:38.171115 (+ 391us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:38.171170 (+ 55us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":152374,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":1023,"apply.run_cpu_time_us":3395,"apply.run_wall_time_us":14233,"num_ops":2,"prepare.queue_time_us":26253,"prepare.run_cpu_time_us":3745,"prepare.run_wall_time_us":3746,"raft.queue_time_us":497,"raft.run_cpu_time_us":1467,"raft.run_wall_time_us":1466,"replication_time_us":93139}]]}
I20260810 23:32:40.783283 13691 ts_tablet_manager.cc:1428] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.420s user 0.006s sys 0.003s
I20260810 23:32:40.784155 12240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_1e5f07908d444cf9af2cae2ea83ae7ae" } modify_external_catalogs: true
W20260810 23:32:40.778299 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 56830.2us
W20260810 23:32:40.678937 12228 rpcz_store.cc:269] 0810 23:32:37.445291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:37.510689 (+ 65398us) service_pool.cc:227] Handling call
0810 23:32:37.712541 (+201852us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:40.538951 (+2826410us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:38.444895 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:38.445479 (+ 584us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:38.445517 (+ 38us) write_op.cc:435] Acquired schema lock
0810 23:32:38.445536 (+ 19us) tablet.cc:660] Decoding operations
0810 23:32:38.445892 (+ 356us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:38.445951 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:38.445984 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:38.446174 (+ 190us) tablet.cc:699] Row locks acquired
0810 23:32:38.446194 (+ 20us) write_op.cc:260] PREPARE: finished
0810 23:32:38.446351 (+ 157us) write_op.cc:270] Start()
0810 23:32:38.446463 (+ 112us) write_op.cc:276] Timestamp: P: 1786404758446248 usec, L: 0
0810 23:32:38.446484 (+ 21us) op_driver.cc:331] REPLICATION: starting
0810 23:32:38.447197 (+ 713us) log.cc:842] Serialized 356 byte log entry
0810 23:32:38.459533 (+ 12336us) op_driver.cc:444] REPLICATION: finished
0810 23:32:38.459987 (+ 454us) write_op.cc:301] APPLY: starting
0810 23:32:38.460080 (+ 93us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:38.460211 (+ 131us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:38.460344 (+ 133us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:38.460846 (+ 502us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:38.461034 (+ 188us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:38.461096 (+ 62us) write_op.cc:312] APPLY: finished
0810 23:32:38.461352 (+ 256us) log.cc:842] Serialized 23 byte log entry
0810 23:32:38.461534 (+ 182us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:38.461631 (+ 97us) write_op.cc:454] Released schema lock
0810 23:32:38.461672 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:38.461901 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:38.461937 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":149306,"reported_tablets":13,"spinlock_wait_cycles":444734848,"child_traces":[["op",{"apply.queue_time_us":258,"apply.run_cpu_time_us":1910,"apply.run_wall_time_us":2145,"num_ops":1,"prepare.queue_time_us":138526,"prepare.run_cpu_time_us":2696,"prepare.run_wall_time_us":2697,"raft.queue_time_us":1370,"raft.run_cpu_time_us":862,"raft.run_wall_time_us":862,"replication_time_us":12944}]]}
I20260810 23:32:40.788801 13691 tablet_bootstrap.cc:492] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:40.789502 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_d36db682741e4ef6bde7416df55ab535"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:40.791090 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d36db682741e4ef6bde7416df55ab535 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:40.600274 12223 rpcz_store.cc:269] 0810 23:32:35.397312 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.522730 (+125418us) service_pool.cc:227] Handling call
0810 23:32:40.413111 (+4890381us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:37.047931 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:37.064431 (+ 16500us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:37.064504 (+ 73us) write_op.cc:435] Acquired schema lock
0810 23:32:37.064555 (+ 51us) tablet.cc:660] Decoding operations
0810 23:32:37.065505 (+ 950us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:37.065648 (+ 143us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:37.065699 (+ 51us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:37.066049 (+ 350us) tablet.cc:699] Row locks acquired
0810 23:32:37.066071 (+ 22us) write_op.cc:260] PREPARE: finished
0810 23:32:37.066448 (+ 377us) write_op.cc:270] Start()
0810 23:32:37.066681 (+ 233us) write_op.cc:276] Timestamp: P: 1786404757066350 usec, L: 0
0810 23:32:37.066704 (+ 23us) op_driver.cc:331] REPLICATION: starting
0810 23:32:37.126683 (+ 59979us) log.cc:842] Serialized 536 byte log entry
0810 23:32:37.251910 (+125227us) op_driver.cc:444] REPLICATION: finished
0810 23:32:37.252581 (+ 671us) write_op.cc:301] APPLY: starting
0810 23:32:37.252711 (+ 130us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:37.252895 (+ 184us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:37.253064 (+ 169us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:37.253385 (+ 321us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:37.253707 (+ 322us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:37.253859 (+ 152us) write_op.cc:312] APPLY: finished
0810 23:32:37.254155 (+ 296us) log.cc:842] Serialized 23 byte log entry
0810 23:32:37.254433 (+ 278us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:37.254541 (+ 108us) write_op.cc:454] Released schema lock
0810 23:32:37.254608 (+ 67us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:37.254907 (+ 299us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:37.254931 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":403,"apply.run_cpu_time_us":2644,"apply.run_wall_time_us":2643,"num_ops":1,"prepare.queue_time_us":115024,"prepare.run_cpu_time_us":6166,"prepare.run_wall_time_us":81106,"raft.queue_time_us":22605,"raft.run_cpu_time_us":1034,"raft.run_wall_time_us":6838,"replication_time_us":185069,"spinlock_wait_cycles":706432}]]}
I20260810 23:32:40.792677 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:40.784439 13665 raft_consensus.cc:348] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:40.797511 12224 catalog_manager.cc:6565] ReplaceTablet: tablet 42517f11422b42c39db282e2c5f584a3 deleted and replaced by tablet a480093c481941f0bb35949c7606f37f
I20260810 23:32:40.784734 12240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_1e5f07908d444cf9af2cae2ea83ae7ae" } modify_external_catalogs: true
I20260810 23:32:40.800302 13008 tablet_service.cc:1514] Processing DeleteTablet for tablet 42517f11422b42c39db282e2c5f584a3 with delete_type TABLET_DATA_DELETED (replaced by tablet a480093c481941f0bb35949c7606f37f) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:40.715528 12229 rpcz_store.cc:269] 0810 23:32:36.662681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:36.673723 (+ 11042us) service_pool.cc:227] Handling call
0810 23:32:36.878880 (+205157us) spinlock_profiling.cc:243] Waited 48.8 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:40.505202 (+3626322us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3252,"reported_tablets":16,"spinlock_wait_cycles":112353024}
I20260810 23:32:40.794585 13665 raft_consensus.cc:374] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:40.812767 13665 raft_consensus.cc:729] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:40.813745 13665 consensus_queue.cc:260] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:40.822428 13665 ts_tablet_manager.cc:1428] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.051s user 0.003s sys 0.006s
I20260810 23:32:40.823659 13665 tablet_bootstrap.cc:492] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
W20260810 23:32:40.826504 12215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260810 23:32:40.828645 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet b8986e59b2314bd69b9b1de36f1f2ba7 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d697e7533af4fc0a15cb41f3fcce4db) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:40.680038 12958 leader_election.cc:302] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:40.833951 12745 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:47536 (request call id 61) took 1320 ms. Trace:
I20260810 23:32:40.834684 12745 rpcz_store.cc:276] 0810 23:32:39.513117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:39.515450 (+ 2333us) service_pool.cc:227] Handling call
0810 23:32:40.833810 (+1318360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:40.840595 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b29cda94584badb1225477ec2bd1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:33 UTC) from {username='slave'} at 127.0.0.1:47546
W20260810 23:32:40.843355 12237 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd77526f1db4455291fde8a001092a09 consensus state has no leader
I20260810 23:32:40.848722 13974 ts_tablet_manager.cc:1918] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:40.851322 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (DEFAULT_TABLE table=default.Table_34eb8fd429274b5d8af145c70aab107b [id=dae244793fd24816a8199c38f23a25b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.852983 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:40.852787 12215 consensus_peers.cc:487] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 -> Peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Couldn't send request to peer 198f534261994047a68e1d509bb2fd8b. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260810 23:32:40.858183 12229 master_service.cc:946] ReplaceTablet: received request to replace tablet 133654008f5247b2ac87dcd09195f197 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:40.860550 13974 log.cc:1196] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/8d2f8688fc6046d58009800c5e288302
I20260810 23:32:40.870870 12767 tablet_service.cc:1467] Processing CreateTablet for tablet 240adfa4be844518bbc9624d0e408789 (DEFAULT_TABLE table=default.Table_32d7e9316d054deca361c6e3d9b5e1f4 [id=43ef82e91d4a4e2b8bb06d4cab61c8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.851742 14145 raft_consensus.cc:2726] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:32:40.873864 12767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240adfa4be844518bbc9624d0e408789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.876562 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:40.878494 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:40.878276 13665 tablet_bootstrap.cc:654] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:40.893584 13974 ts_tablet_manager.cc:1939] T 8d2f8688fc6046d58009800c5e288302 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:40.895339 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b (DEFAULT_TABLE table=default.Table_d942bf134f9849718bb9eb12c24806a1 [id=ff8088bdfc9f456683dd802d62ae67a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.896927 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:40.866389 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 187073us
I20260810 23:32:40.901017 12958 leader_election.cc:302] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:40.908324 14145 raft_consensus.cc:2781] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260810 23:32:40.908660 12700 leader_election.cc:341] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:40.908957 14145 raft_consensus.cc:482] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:40.909386 14145 raft_consensus.cc:3037] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:40.909638 12700 leader_election.cc:302] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:40.910570 14178 raft_consensus.cc:482] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:32:40.884506 12218 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet bd77526f1db4455291fde8a001092a09 (table default.Table_7ed38c2d149c496393381b8394b2ee06 [id=963512beef0d4063b8e6a5738abd1d92], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bd77526f1db4455291fde8a001092a09 consensus state has no leader
I20260810 23:32:40.893570 12235 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 185) took 1513 ms. Trace:
I20260810 23:32:40.910846 14187 raft_consensus.cc:2726] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:40.911185 14178 raft_consensus.cc:504] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:40.911214 12235 rpcz_store.cc:276] 0810 23:32:39.379967 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.689699 (+1309732us) service_pool.cc:227] Handling call
0810 23:32:40.723923 (+ 34224us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7156 00007f0ce6e24034 00007f0ce6e23fc2 00007f0ce6e23ef1 00007f0ce6e23e6c 00007f0ce6e23122 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:40.749545 (+ 25622us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:40.751492 (+ 1947us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:40.892617 (+141125us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:40.893083 (+ 466us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:40.893452 (+ 369us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:40.769007 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:40.830022 (+ 61015us) spinlock_profiling.cc:243] Waited 4.88 ms on lock 0x7b5400031528. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce97b3e91 00007f0ce97b2afc 00007f0ce7788a4d 00007f0ce7774603 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784
0810 23:32:40.830081 (+ 59us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:40.830104 (+ 23us) write_op.cc:435] Acquired schema lock
0810 23:32:40.830117 (+ 13us) tablet.cc:660] Decoding operations
0810 23:32:40.830434 (+ 317us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:40.830479 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:40.830501 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:40.830708 (+ 207us) tablet.cc:699] Row locks acquired
0810 23:32:40.830721 (+ 13us) write_op.cc:260] PREPARE: finished
0810 23:32:40.830880 (+ 159us) write_op.cc:270] Start()
0810 23:32:40.831145 (+ 265us) write_op.cc:276] Timestamp: P: 1786404760830811 usec, L: 0
0810 23:32:40.831166 (+ 21us) op_driver.cc:331] REPLICATION: starting
0810 23:32:40.831764 (+ 598us) log.cc:842] Serialized 433 byte log entry
0810 23:32:40.886694 (+ 54930us) op_driver.cc:444] REPLICATION: finished
0810 23:32:40.887578 (+ 884us) write_op.cc:301] APPLY: starting
0810 23:32:40.887668 (+ 90us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:40.887817 (+ 149us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:40.887848 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:40.888447 (+ 599us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:40.888671 (+ 224us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:40.888719 (+ 48us) write_op.cc:312] APPLY: finished
0810 23:32:40.889088 (+ 369us) log.cc:842] Serialized 29 byte log entry
0810 23:32:40.889333 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:40.889473 (+ 140us) write_op.cc:454] Released schema lock
0810 23:32:40.889531 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:40.889849 (+ 318us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:40.889900 (+ 51us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":3062016,"child_traces":[["op",{"apply.queue_time_us":625,"apply.run_cpu_time_us":2565,"apply.run_wall_time_us":2566,"num_ops":2,"prepare.queue_time_us":16176,"prepare.run_cpu_time_us":7813,"prepare.run_wall_time_us":64894,"raft.queue_time_us":115,"raft.run_cpu_time_us":1421,"raft.run_wall_time_us":1423,"replication_time_us":55408,"spinlock_wait_cycles":11229056}]]}
I20260810 23:32:40.909845 12236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 124259164adf404c9cbe5f4e0d5e16c0
I20260810 23:32:40.917426 12236 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 186) took 1260 ms. Trace:
I20260810 23:32:40.917429 14178 leader_election.cc:288] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:40.917872 12236 rpcz_store.cc:276] 0810 23:32:39.657030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.690722 (+1033692us) service_pool.cc:227] Handling call
0810 23:32:40.726831 (+ 36109us) spinlock_profiling.cc:243] Waited 4.57 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7466 00007f0ce6e28b34 00007f0ce6e28ac2 00007f0ce6e289f1 00007f0ce6e2896c 00007f0ce6e27c22 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:40.728221 (+ 1390us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:40.728826 (+ 605us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:40.729022 (+ 196us) catalog_manager.cc:2761] Locking tablets
0810 23:32:40.729960 (+ 938us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:40.909229 (+179269us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:40.909564 (+ 335us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:40.909721 (+ 157us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:40.917359 (+ 7638us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:40.834035 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:40.834507 (+ 472us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:40.834531 (+ 24us) write_op.cc:435] Acquired schema lock
0810 23:32:40.834543 (+ 12us) tablet.cc:660] Decoding operations
0810 23:32:40.834859 (+ 316us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:40.834896 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:40.834919 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:40.835287 (+ 368us) tablet.cc:699] Row locks acquired
0810 23:32:40.835301 (+ 14us) write_op.cc:260] PREPARE: finished
0810 23:32:40.835430 (+ 129us) write_op.cc:270] Start()
0810 23:32:40.835534 (+ 104us) write_op.cc:276] Timestamp: P: 1786404760835365 usec, L: 0
0810 23:32:40.835561 (+ 27us) op_driver.cc:331] REPLICATION: starting
0810 23:32:40.885709 (+ 50148us) log.cc:842] Serialized 734 byte log entry
0810 23:32:40.900626 (+ 14917us) op_driver.cc:444] REPLICATION: finished
0810 23:32:40.901342 (+ 716us) write_op.cc:301] APPLY: starting
0810 23:32:40.901488 (+ 146us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:40.901720 (+ 232us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:40.901875 (+ 155us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:40.902267 (+ 392us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:40.902656 (+ 389us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:40.902751 (+ 95us) write_op.cc:312] APPLY: finished
0810 23:32:40.903133 (+ 382us) log.cc:842] Serialized 29 byte log entry
0810 23:32:40.904037 (+ 904us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:40.904156 (+ 119us) write_op.cc:454] Released schema lock
0810 23:32:40.904212 (+ 56us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:40.904576 (+ 364us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:40.904624 (+ 48us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":10504448,"child_traces":[["op",{"apply.queue_time_us":472,"apply.run_cpu_time_us":2977,"apply.run_wall_time_us":3549,"mutex_wait_us":8408,"num_ops":2,"prepare.queue_time_us":46144,"prepare.run_cpu_time_us":2931,"prepare.run_wall_time_us":81499,"raft.queue_time_us":8626,"raft.run_cpu_time_us":22,"raft.run_wall_time_us":22,"replication_time_us":64931}]]}
W20260810 23:32:40.928197 14180 log.cc:924] Time spent T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.053s user 0.002s sys 0.000s
I20260810 23:32:40.930017 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_d239f0c3798a49c1bf1055dab20c108a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:40.932286 12235 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_9a86e4c2641044419dffb5dbef58e581" } new_table_name: "default.Table_e0faeb2fda9f4af48212613473c88681" modify_external_catalogs: true
I20260810 23:32:40.942214 12763 tablet_service.cc:1467] Processing CreateTablet for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (DEFAULT_TABLE table=default.Table_d6f432a771c544d6acfb28ec150340da [id=2c8320255aab4036b5350d082f9e04ef]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:40.942863 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d239f0c3798a49c1bf1055dab20c108a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:40.948549 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7bd9b6ca4845d8b8705ff970c8f06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.949533 14121 ts_tablet_manager.cc:1918] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:40.950407 14121 log.cc:1196] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/8b1f7c16b6bc4dc0bac80ac55332f056
I20260810 23:32:40.956965 14121 ts_tablet_manager.cc:1939] T 8b1f7c16b6bc4dc0bac80ac55332f056 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:40.958516 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 104) took 2056 ms. Trace:
I20260810 23:32:40.974341 12769 tablet_service.cc:1467] Processing CreateTablet for tablet a41ddf9f6f71443ba75c97d2b010372c (DEFAULT_TABLE table=default.Table_73f2b1b207fb45f2adecbd37412c5e6f [id=c57f7c273a9841c1ac02523a733a383f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.977931 12764 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (DEFAULT_TABLE table=default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:40.981580 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41ddf9f6f71443ba75c97d2b010372c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.982542 13691 tablet_bootstrap.cc:654] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:40.984758 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1b9d20d09c4d1ba9a4b10959807e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.979988 14170 raft_consensus.cc:482] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:40.986670 14170 raft_consensus.cc:504] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:40.989874 14170 leader_election.cc:288] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:40.991852 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 124259164adf404c9cbe5f4e0d5e16c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:41.002159 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 108) took 2334 ms. Trace:
I20260810 23:32:41.003912 12757 tablet_service.cc:1514] Processing DeleteTablet for tablet 081b4e1f53c74f768a4bf401ae22b585 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:32 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:41.029860 13935 ts_tablet_manager.cc:1918] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:41.037338 12234 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 184) took 1716 ms. Trace:
W20260810 23:32:41.046638 12958 leader_election.cc:341] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 840400119d20424ea0f03f2bff550d44
I20260810 23:32:41.047696 14136 consensus_queue.cc:1035] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:41.032994 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 62705f99687340aaa1aca0942b28487f (DEFAULT_TABLE table=default.Table_96019e28037b4d38aa9fc933a29c8884 [id=dd7ca129aac0430ba386c53a30052148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:41.055092 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62705f99687340aaa1aca0942b28487f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:41.040417 13514 tablet_bootstrap.cc:654] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:41.042042 12760 tablet_service.cc:1467] Processing CreateTablet for tablet 684caf5aaa8347e8b4e02323318fb0b7 (DEFAULT_TABLE table=default.Table_d4fad7cecea149e88811df34c8c11d2d [id=997d599555454bbca33e0b1d98f2f88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:41.065068 12760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 684caf5aaa8347e8b4e02323318fb0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:41.051218 12766 tablet_service.cc:1514] Processing DeleteTablet for tablet 42517f11422b42c39db282e2c5f584a3 with delete_type TABLET_DATA_DELETED (replaced by tablet a480093c481941f0bb35949c7606f37f) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:41.040524 12234 rpcz_store.cc:276] 0810 23:32:39.321165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.688263 (+1367098us) service_pool.cc:227] Handling call
0810 23:32:40.757518 (+ 69255us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:40.759043 (+ 1525us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:41.036417 (+277374us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:41.036874 (+ 457us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:41.037228 (+ 354us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:40.915654 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:40.916101 (+ 447us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:40.916128 (+ 27us) write_op.cc:435] Acquired schema lock
0810 23:32:40.916142 (+ 14us) tablet.cc:660] Decoding operations
0810 23:32:40.916448 (+ 306us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:40.916494 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:40.916517 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:40.916732 (+ 215us) tablet.cc:699] Row locks acquired
0810 23:32:40.916748 (+ 16us) write_op.cc:260] PREPARE: finished
0810 23:32:40.916890 (+ 142us) write_op.cc:270] Start()
0810 23:32:40.916987 (+ 97us) write_op.cc:276] Timestamp: P: 1786404760916821 usec, L: 0
0810 23:32:40.917006 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 23:32:40.917716 (+ 710us) log.cc:842] Serialized 433 byte log entry
0810 23:32:41.030895 (+113179us) op_driver.cc:444] REPLICATION: finished
0810 23:32:41.032250 (+ 1355us) write_op.cc:301] APPLY: starting
0810 23:32:41.032327 (+ 77us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:41.032562 (+ 235us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:41.032669 (+ 107us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:41.033391 (+ 722us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:41.033607 (+ 216us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:41.033657 (+ 50us) write_op.cc:312] APPLY: finished
0810 23:32:41.033990 (+ 333us) log.cc:842] Serialized 29 byte log entry
0810 23:32:41.034279 (+ 289us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:41.034421 (+ 142us) write_op.cc:454] Released schema lock
0810 23:32:41.034481 (+ 60us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:41.034820 (+ 339us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:41.034874 (+ 54us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1066,"apply.run_cpu_time_us":2918,"apply.run_wall_time_us":2918,"num_ops":2,"prepare.queue_time_us":90035,"prepare.run_cpu_time_us":2526,"prepare.run_wall_time_us":20878,"raft.queue_time_us":8816,"raft.run_cpu_time_us":17,"raft.run_wall_time_us":17,"replication_time_us":113726}]]}
W20260810 23:32:41.074146 12700 leader_election.cc:341] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:41.074662 12700 leader_election.cc:302] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:41.103453 12234 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_bacbea393b664097a0b52af2ce5292de" } new_table_name: "default.Table_12334169bd044f07914979e00a62582d" modify_external_catalogs: true
I20260810 23:32:41.125135 12240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet ca0dd19112504f8ca57f64dcf0bd50a8
I20260810 23:32:41.130769 13007 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0dd19112504f8ca57f64dcf0bd50a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:41.134177 14170 raft_consensus.cc:482] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:41.134958 14170 raft_consensus.cc:504] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:41.138159 14170 leader_election.cc:288] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:41.132601 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b261673faa964b65af3576c89e302864" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:41.139364 12229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 133654008f5247b2ac87dcd09195f197
I20260810 23:32:40.959087 14121 rpcz_store.cc:276] 0810 23:32:38.902168 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.902454 (+ 286us) service_pool.cc:227] Handling call
0810 23:32:39.157079 (+254625us) ts_tablet_manager.cc:1145] Deleting tablet 8b1f7c16b6bc4dc0bac80ac55332f056
0810 23:32:39.389556 (+232477us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:40.298035 (+908479us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:40.948629 (+650594us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:40.958397 (+ 9768us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":160593,"spinlock_wait_cycles":57216,"tablet-delete.queue_time_us":253125,"thread_start_us":237671,"threads_started":1}
I20260810 23:32:41.141886 14121 tablet_replica.cc:330] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:41.143167 12229 catalog_manager.cc:6565] ReplaceTablet: tablet 133654008f5247b2ac87dcd09195f197 deleted and replaced by tablet ca8c0d1dacc344cbacb0c5988f0e2221
I20260810 23:32:41.143149 14121 raft_consensus.cc:2229] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:41.143922 14121 raft_consensus.cc:2258] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:41.144802 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 8b1f7c16b6bc4dc0bac80ac55332f056 (table default.Table_e6944762a73d4c80ae0866fad8945b41 [id=50e2197e7c13449caeebf39a42ba347c]) successfully deleted
W20260810 23:32:41.146078 12235 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92b81e7056a441d2854ba5ab6b4c5433 consensus state has no leader
I20260810 23:32:41.149237 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ba75f2c2b54ec1968db8feeacf8383" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:41.150003 12802 raft_consensus.cc:2454] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:41.151605 14178 raft_consensus.cc:2726] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:41.151582 12235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_514d038dba3a4fc399b97e7edc254ad9" } modify_external_catalogs: true
I20260810 23:32:41.152298 12235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_514d038dba3a4fc399b97e7edc254ad9" } modify_external_catalogs: true
I20260810 23:32:41.155444 12229 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_22eeebd76f0e44bca7287cafea0de6f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:41.157071 12229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22eeebd76f0e44bca7287cafea0de6f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:41.159925 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 85827.2us
I20260810 23:32:41.163457 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet 133654008f5247b2ac87dcd09195f197 with delete_type TABLET_DATA_DELETED (replaced by tablet ca8c0d1dacc344cbacb0c5988f0e2221) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:41.165527 12700 leader_election.cc:341] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:41.107745 13665 tablet_bootstrap.cc:492] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:41.168787 13665 ts_tablet_manager.cc:1397] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.345s user 0.011s sys 0.011s
I20260810 23:32:41.110957 14041 raft_consensus.cc:482] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:41.172430 14041 raft_consensus.cc:504] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:41.176440 13665 raft_consensus.cc:348] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:41.177812 13665 raft_consensus.cc:374] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:41.178269 13665 raft_consensus.cc:729] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:41.179172 13665 consensus_queue.cc:260] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:41.182483 13665 ts_tablet_manager.cc:1428] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260810 23:32:41.157408 12224 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 874ca0c3586b41a8a3074d8988813633 (table default.Table_b57f0f6e94384d22af5d3bf7817d5044 [id=4bb9a6fbe6e644c583a468af86950cef]). Expected version 1 got 0
I20260810 23:32:41.182864 14041 leader_election.cc:288] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:41.184695 12224 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:41.186124 12224 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_75ca982c15e34c8d9603f100dff5b392 [id=b809029bf6cc48ab917e48bbf2109ae1]). Expected version 2 got 0
I20260810 23:32:41.186677 13665 tablet_bootstrap.cc:492] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:41.188037 13935 log.cc:1196] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/dd83bbe7a1664c278098fa66b9c6895e
I20260810 23:32:41.196734 12958 leader_election.cc:302] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:41.197682 14170 consensus_queue.cc:1035] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:41.197880 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 124259164adf404c9cbe5f4e0d5e16c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:41.199549 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 3d596299c82b4ff1ac151d68b97ff55b (table default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]) successfully deleted
W20260810 23:32:41.200861 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet b261673faa964b65af3576c89e302864 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b261673faa964b65af3576c89e302864 already in progress: creating tablet
W20260810 23:32:41.201527 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet db363eb0f98d460eb6f5a577233d8ac0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db363eb0f98d460eb6f5a577233d8ac0 already in progress: creating tablet
I20260810 23:32:41.202628 14136 raft_consensus.cc:2781] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:41.203217 14136 raft_consensus.cc:482] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:41.203593 14136 raft_consensus.cc:3037] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:41.207620 13935 ts_tablet_manager.cc:1939] T dd83bbe7a1664c278098fa66b9c6895e P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:41.210238 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 105) took 2887 ms. Trace:
I20260810 23:32:41.210624 13935 rpcz_store.cc:276] 0810 23:32:38.323056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.326929 (+ 3873us) service_pool.cc:227] Handling call
0810 23:32:40.531048 (+2204119us) ts_tablet_manager.cc:1145] Deleting tablet dd83bbe7a1664c278098fa66b9c6895e
0810 23:32:40.531088 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:40.772011 (+240923us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:41.029540 (+257529us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:41.210158 (+180618us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45821,"tablet-delete.queue_time_us":2202683}
I20260810 23:32:41.192412 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840400119d20424ea0f03f2bff550d44" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:41.213200 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ba75f2c2b54ec1968db8feeacf8383" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:41.183574 13974 rpcz_store.cc:276] 0810 23:32:38.667423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.670095 (+ 2672us) service_pool.cc:227] Handling call
0810 23:32:40.277638 (+1607543us) ts_tablet_manager.cc:1145] Deleting tablet 8d2f8688fc6046d58009800c5e288302
0810 23:32:40.277692 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:40.473420 (+195728us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:40.747967 (+274547us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:41.001983 (+254016us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1606706}
I20260810 23:32:41.214067 13336 raft_consensus.cc:2454] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:41.224933 14121 ts_tablet_manager.cc:1905] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:41.225472 12952 leader_election.cc:341] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:41.225952 12952 leader_election.cc:302] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:41.230041 13691 tablet_bootstrap.cc:492] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:41.231312 13691 ts_tablet_manager.cc:1397] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.446s user 0.012s sys 0.009s
I20260810 23:32:41.234553 13974 tablet_replica.cc:330] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:41.235366 14145 raft_consensus.cc:504] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:41.260290 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_6aa94428c3854c10bbb554fff1d855a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:41.262269 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6aa94428c3854c10bbb554fff1d855a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:41.285554 13514 tablet_bootstrap.cc:492] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:41.286121 13514 ts_tablet_manager.cc:1397] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.507s user 0.013s sys 0.002s
W20260810 23:32:41.286265 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 60858.7us
I20260810 23:32:41.306433 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0dd19112504f8ca57f64dcf0bd50a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:41.307966 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 133654008f5247b2ac87dcd09195f197 with delete_type TABLET_DATA_DELETED (replaced by tablet ca8c0d1dacc344cbacb0c5988f0e2221) from {username='slave'} at 127.0.0.1:47546
W20260810 23:32:41.309002 13018 consensus_meta.cc:287] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.519s user 0.010s sys 0.003s
I20260810 23:32:41.309247 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet dd83bbe7a1664c278098fa66b9c6895e (table default.Table_7c223ce60be04a498d08155325785819 [id=0eeab77fa3d54892a91f87f2fbdd673d]) successfully deleted
I20260810 23:32:41.317000 14224 raft_consensus.cc:482] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:41.330655 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_848da5bb390440608b71e81db171df2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:41.333074 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_848da5bb390440608b71e81db171df2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:41.321348 14218 raft_consensus.cc:2726] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:41.345796 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b261673faa964b65af3576c89e302864" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:41.350364 12698 leader_election.cc:341] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:41.350306 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b29cda94584badb1225477ec2bd1bc" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:41.350945 12698 leader_election.cc:302] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:41.351815 13691 raft_consensus.cc:348] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:41.352308 14178 raft_consensus.cc:2726] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:41.352619 13691 raft_consensus.cc:374] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:41.352897 13691 raft_consensus.cc:729] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:41.360652 14145 leader_election.cc:288] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:41.348253 14136 raft_consensus.cc:504] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:41.373311 13286 consensus_meta.cc:287] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent flushing consensus metadata: real 0.529s user 0.014s sys 0.000s
I20260810 23:32:41.375177 14136 leader_election.cc:288] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:41.353561 13691 consensus_queue.cc:260] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:41.378392 13691 ts_tablet_manager.cc:1428] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.145s user 0.006s sys 0.000s
I20260810 23:32:41.379215 13212 leader_election.cc:302] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:41.380661 14206 raft_consensus.cc:2781] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:32:41.381011 14206 raft_consensus.cc:482] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:41.381356 14206 raft_consensus.cc:3037] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:32:41.363443 14224 raft_consensus.cc:504] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:41.391948 14224 leader_election.cc:288] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:41.403064 13691 tablet_bootstrap.cc:492] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:41.403309 12236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_73f2b1b207fb45f2adecbd37412c5e6f" } modify_external_catalogs: true
I20260810 23:32:41.211874 13935 tablet_replica.cc:330] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:41.404215 12236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_73f2b1b207fb45f2adecbd37412c5e6f" } modify_external_catalogs: true
I20260810 23:32:41.406158 12223 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_085f24edadf34deea7a3d9356d00c7b1" } new_table_name: "default.Table_bf4e391ee20b45d393a269203b401bc0" modify_external_catalogs: true
I20260810 23:32:41.411144 13334 raft_consensus.cc:3037] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:41.419040 13514 raft_consensus.cc:348] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:41.427420 14203 raft_consensus.cc:482] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:41.428243 14203 raft_consensus.cc:504] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:41.431008 14203 leader_election.cc:288] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
W20260810 23:32:41.451850 12223 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d0d9227d8574de8bd73041f3e8043a0 consensus state has no leader
I20260810 23:32:41.485029 13022 tablet_service.cc:1467] Processing CreateTablet for tablet a480093c481941f0bb35949c7606f37f (DEFAULT_TABLE table=default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:41.486579 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a480093c481941f0bb35949c7606f37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:41.488808 13331 raft_consensus.cc:3037] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:41.495241 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
I20260810 23:32:41.494925 13333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e26fe4d42445279f12bb17b2841d08" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:41.504834 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:41.505692 13691 tablet_bootstrap.cc:654] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:41.506186 14236 consensus_queue.cc:1035] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260810 23:32:41.512427 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874ca0c3586b41a8a3074d8988813633" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
W20260810 23:32:41.516152 12952 leader_election.cc:341] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:41.528612 12236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet a41ddf9f6f71443ba75c97d2b010372c
I20260810 23:32:41.534746 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet a41ddf9f6f71443ba75c97d2b010372c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:41.536653 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet bd77526f1db4455291fde8a001092a09 (table default.Table_7ed38c2d149c496393381b8394b2ee06 [id=963512beef0d4063b8e6a5738abd1d92]). Expected version 1 got 0
I20260810 23:32:41.537652 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 874ca0c3586b41a8a3074d8988813633 (table default.Table_b57f0f6e94384d22af5d3bf7817d5044 [id=4bb9a6fbe6e644c583a468af86950cef]). Expected version 1 got 0
I20260810 23:32:41.540205 12240 catalog_manager.cc:5495] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129). New cstate: current_term: 1 leader_uuid: "191d120f4c7f44fc8475305a6d2e75f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:41.542834 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:41.549950 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_6b8e1ec40cdc4eda823368ca59437332"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:41.558353 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b8e1ec40cdc4eda823368ca59437332 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:41.519343 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ba75f2c2b54ec1968db8feeacf8383" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:41.559782 13338 raft_consensus.cc:3037] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:41.565832 12226 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_75ca982c15e34c8d9603f100dff5b392 [id=b809029bf6cc48ab917e48bbf2109ae1] alter complete (version 2)
W20260810 23:32:41.573817 12226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 20) took 1162 ms (client timeout 1000 ms). Trace:
W20260810 23:32:41.574524 12226 rpcz_store.cc:269] 0810 23:32:40.411179 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.683017 (+271838us) service_pool.cc:227] Handling call
0810 23:32:41.573729 (+890712us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:41.154575 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:41.154981 (+ 406us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:41.155003 (+ 22us) write_op.cc:435] Acquired schema lock
0810 23:32:41.155016 (+ 13us) tablet.cc:660] Decoding operations
0810 23:32:41.155298 (+ 282us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:41.155337 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:41.155358 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:41.155536 (+ 178us) tablet.cc:699] Row locks acquired
0810 23:32:41.155550 (+ 14us) write_op.cc:260] PREPARE: finished
0810 23:32:41.155680 (+ 130us) write_op.cc:270] Start()
0810 23:32:41.155769 (+ 89us) write_op.cc:276] Timestamp: P: 1786404761155627 usec, L: 0
0810 23:32:41.155787 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 23:32:41.156348 (+ 561us) log.cc:842] Serialized 356 byte log entry
0810 23:32:41.338917 (+182569us) op_driver.cc:444] REPLICATION: finished
0810 23:32:41.339178 (+ 261us) write_op.cc:301] APPLY: starting
0810 23:32:41.339231 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:41.339333 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:41.339433 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:41.339610 (+ 177us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:41.339742 (+ 132us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:41.339782 (+ 40us) write_op.cc:312] APPLY: finished
0810 23:32:41.339961 (+ 179us) log.cc:842] Serialized 23 byte log entry
0810 23:32:41.340137 (+ 176us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:41.340204 (+ 67us) write_op.cc:454] Released schema lock
0810 23:32:41.340240 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:41.411291 (+ 71051us) spinlock_profiling.cc:243] Waited 70.4 ms on lock 0x7b5400031480. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce76ac361 00007f0ce777d254 00007f0ce777686b 00007f0ce7775fd1 00007f0ce7779dd1 00007f0ce7779d89 00007f0ce7779d19 00007f0ce7779ce1 00007f0ce7778fdd 0000000000580784 00000000005805b9
0810 23:32:41.411362 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:41.411384 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":98,"apply.run_cpu_time_us":3261,"apply.run_wall_time_us":72423,"num_ops":1,"prepare.queue_time_us":110,"prepare.run_cpu_time_us":2086,"prepare.run_wall_time_us":2096,"raft.queue_time_us":1335,"raft.run_cpu_time_us":25,"raft.run_wall_time_us":27,"replication_time_us":183052,"spinlock_wait_cycles":162017152}]]}
W20260810 23:32:41.580729 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 283695us
I20260810 23:32:41.588260 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:30 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:41.590795 13514 raft_consensus.cc:374] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:41.591126 13514 raft_consensus.cc:729] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:41.591792 13514 consensus_queue.cc:260] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:41.466688 13935 raft_consensus.cc:2229] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:41.523134 12756 tablet_service.cc:1467] Processing CreateTablet for tablet a480093c481941f0bb35949c7606f37f (DEFAULT_TABLE table=default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:41.557088 12768 tablet_service.cc:1514] Processing DeleteTablet for tablet a41ddf9f6f71443ba75c97d2b010372c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:47546
W20260810 23:32:41.582801 12700 consensus_peers.cc:487] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3 -> Peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Couldn't send request to peer f26517150f1c480a877f15aec71d020b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:41.600454 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b29cda94584badb1225477ec2bd1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:33 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:41.614553 13665 tablet_bootstrap.cc:654] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:41.616734 13691 tablet_bootstrap.cc:492] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:41.617518 13691 ts_tablet_manager.cc:1397] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.215s user 0.019s sys 0.003s
I20260810 23:32:41.625108 13935 raft_consensus.cc:2258] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:41.628820 13935 ts_tablet_manager.cc:1905] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:41.630319 13031 consensus_meta.cc:287] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.613s user 0.009s sys 0.000s
I20260810 23:32:41.636857 12235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 423e8d583d704fa9834026ebcd57e17e
I20260810 23:32:41.642823 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet 423e8d583d704fa9834026ebcd57e17e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:41.625625 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_9a2539ffd1684a4cae73196a0ab97172"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:41.647688 13014 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d596299c82b4ff1ac151d68b97ff55b with delete_type TABLET_DATA_DELETED (replaced by tablet 4cdec13ee70d402fa3a08119e7bf1313) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:41.648468 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a2539ffd1684a4cae73196a0ab97172 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:41.648885 13213 leader_election.cc:341] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 3d596299c82b4ff1ac151d68b97ff55b
I20260810 23:32:41.636350 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ba75f2c2b54ec1968db8feeacf8383" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:41.652709 12803 raft_consensus.cc:3037] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:41.652858 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet b8986e59b2314bd69b9b1de36f1f2ba7 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d697e7533af4fc0a15cb41f3fcce4db) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:41.637323 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874ca0c3586b41a8a3074d8988813633" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:41.654006 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_2553fc7eafdd478a88b6d45dc4dfd846"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:41.656592 12805 raft_consensus.cc:3037] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:41.657536 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2553fc7eafdd478a88b6d45dc4dfd846 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:41.659605 12224 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 27) took 850 ms (client timeout 1000 ms). Trace:
I20260810 23:32:41.659786 12234 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_6aa94428c3854c10bbb554fff1d855a0" } new_table_name: "default.Table_07f851c929404a589a8572028206f242" modify_external_catalogs: true
I20260810 23:32:41.639209 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a480093c481941f0bb35949c7606f37f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:41.659999 12224 rpcz_store.cc:269] 0810 23:32:40.809054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.819102 (+ 10048us) service_pool.cc:227] Handling call
0810 23:32:41.659537 (+840435us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":18}
W20260810 23:32:41.627869 12770 consensus_meta.cc:287] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.589s user 0.010s sys 0.000s
W20260810 23:32:41.685761 14237 log.cc:924] Time spent T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.095s user 0.000s sys 0.002s
I20260810 23:32:41.646631 12768 tablet_service.cc:1514] Processing DeleteTablet for tablet 423e8d583d704fa9834026ebcd57e17e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:41.693351 13031 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 121) took 1120 ms. Trace:
I20260810 23:32:41.694068 13031 rpcz_store.cc:276] 0810 23:32:40.572905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.592182 (+ 19277us) service_pool.cc:227] Handling call
0810 23:32:40.595503 (+ 3321us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:40.595680 (+ 177us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:41.016704 (+421024us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:41.631217 (+614513us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:41.693229 (+ 62012us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:41.711578 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_e1e8440f714f4d55a1622ec3b2f537b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:41.713819 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1e8440f714f4d55a1622ec3b2f537b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:41.731048 13691 raft_consensus.cc:348] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:41.732079 13691 raft_consensus.cc:374] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:41.732388 13691 raft_consensus.cc:729] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:41.733222 13691 consensus_queue.cc:260] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:41.745337 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 97306.9us
I20260810 23:32:41.771564 13514 ts_tablet_manager.cc:1428] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.485s user 0.007s sys 0.003s
I20260810 23:32:41.773211 13514 tablet_bootstrap.cc:492] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:32:41.798120 12958 leader_election.cc:341] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 81e26fe4d42445279f12bb17b2841d08
I20260810 23:32:41.799093 12958 leader_election.cc:302] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:41.615494 14121 ts_tablet_manager.cc:1918] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:41.812644 14121 log.cc:1196] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/dd83bbe7a1664c278098fa66b9c6895e
I20260810 23:32:41.814594 14121 ts_tablet_manager.cc:1939] T dd83bbe7a1664c278098fa66b9c6895e P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:41.728047 14206 raft_consensus.cc:504] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:41.821274 12224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_b57f0f6e94384d22af5d3bf7817d5044" } modify_external_catalogs: true
W20260810 23:32:41.730495 13276 consensus_meta.cc:287] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent flushing consensus metadata: real 0.598s user 0.015s sys 0.002s
I20260810 23:32:41.779808 12805 raft_consensus.cc:2454] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:41.822100 12224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_b57f0f6e94384d22af5d3bf7817d5044" } modify_external_catalogs: true
I20260810 23:32:41.801977 12803 raft_consensus.cc:2454] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
W20260810 23:32:41.808126 12763 consensus_meta.cc:287] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.532s user 0.006s sys 0.007s
I20260810 23:32:41.844350 12237 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_d6f432a771c544d6acfb28ec150340da" } new_table_name: "default.Table_34a45145441d42cbbaf01a336059d4ec" modify_external_catalogs: true
I20260810 23:32:41.875272 12228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_7548a45d866747b9b0944e13b211c643" } modify_external_catalogs: true
W20260810 23:32:41.880508 12234 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1452c292e9c476b9e26f56998f0ad55 has no consensus state
I20260810 23:32:41.880530 12228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_7548a45d866747b9b0944e13b211c643" } modify_external_catalogs: true
W20260810 23:32:41.932530 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f1452c292e9c476b9e26f56998f0ad55 has no consensus state
I20260810 23:32:41.934214 13974 raft_consensus.cc:2229] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260810 23:32:41.972750 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 593740us
I20260810 23:32:41.994796 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_74329e5e7599420e83c0c8646d466e5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:41.997967 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74329e5e7599420e83c0c8646d466e5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:41.827018 13691 ts_tablet_manager.cc:1428] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.209s user 0.006s sys 0.004s
I20260810 23:32:42.000968 13691 tablet_bootstrap.cc:492] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:41.842383 14234 consensus_queue.cc:1035] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260810 23:32:41.844693 14216 raft_consensus.cc:2726] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 23:32:41.845667 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 47628.6us
I20260810 23:32:42.006218 14236 raft_consensus.cc:482] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:42.006754 14236 raft_consensus.cc:504] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:42.014050 14236 leader_election.cc:288] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:42.024394 12768 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d596299c82b4ff1ac151d68b97ff55b with delete_type TABLET_DATA_DELETED (replaced by tablet 4cdec13ee70d402fa3a08119e7bf1313) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:42.029466 14203 consensus_queue.cc:1035] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:41.980203 13974 raft_consensus.cc:2258] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:41.936590 13276 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 133) took 1280 ms. Trace:
I20260810 23:32:42.017764 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046f1a7983ec40489edeaba4d2595b1c" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:42.040071 13276 rpcz_store.cc:276] 0810 23:32:40.656619 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.657854 (+ 1235us) service_pool.cc:227] Handling call
0810 23:32:40.660426 (+ 2572us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:40.660638 (+ 212us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:41.130814 (+470176us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:41.935684 (+804870us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:41.936477 (+ 793us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:42.044093 14206 leader_election.cc:288] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:42.018374 12757 tablet_service.cc:1514] Processing DeleteTablet for tablet dd83bbe7a1664c278098fa66b9c6895e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:36 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:42.055102 12763 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 135) took 1126 ms. Trace:
I20260810 23:32:42.065259 12763 rpcz_store.cc:276] 0810 23:32:40.928955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.940428 (+ 11473us) service_pool.cc:227] Handling call
0810 23:32:40.943545 (+ 3117us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:40.943752 (+ 207us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:41.250243 (+306491us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:42.016609 (+766366us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:42.047458 (+ 30849us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:42.076763 13935 ts_tablet_manager.cc:1918] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260810 23:32:42.078140 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 75701.7us
I20260810 23:32:42.078656 13935 log.cc:1196] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/8d2f8688fc6046d58009800c5e288302
I20260810 23:32:42.079559 13935 ts_tablet_manager.cc:1939] T 8d2f8688fc6046d58009800c5e288302 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:42.079600 12958 leader_election.cc:302] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:42.082686 12958 leader_election.cc:302] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:42.083554 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 106) took 3735 ms. Trace:
I20260810 23:32:42.083698 14234 raft_consensus.cc:2781] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:42.084374 14234 raft_consensus.cc:686] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:42.085589 14234 consensus_queue.cc:237] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:42.085508 13935 rpcz_store.cc:276] 0810 23:32:38.348851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.350748 (+ 1897us) service_pool.cc:227] Handling call
0810 23:32:41.211695 (+2860947us) ts_tablet_manager.cc:1145] Deleting tablet 8d2f8688fc6046d58009800c5e288302
0810 23:32:41.211743 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:41.961719 (+749976us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:42.076454 (+114735us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:42.083485 (+ 7031us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25905,"tablet-delete.queue_time_us":2859670}
I20260810 23:32:42.087661 14216 raft_consensus.cc:2781] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:42.088313 14216 raft_consensus.cc:686] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:42.089243 14216 consensus_queue.cc:237] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:42.095927 13935 tablet_replica.cc:330] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:42.100077 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 8d2f8688fc6046d58009800c5e288302 (table default.Table_783bf3bd81bb4ec294327f9ed5411afb [id=1f49f73dd9d34e968defd3a8ade21d22]) successfully deleted
I20260810 23:32:42.105604 12235 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_2553fc7eafdd478a88b6d45dc4dfd846" } new_table_name: "default.Table_d860efaf3e3a4d1386d4c8e328463a8a" modify_external_catalogs: true
I20260810 23:32:42.111387 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_216b21581f454466a84891c7e3d0e567"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:42.114115 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_216b21581f454466a84891c7e3d0e567 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:42.125118 12237 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff7bd9b6ca4845d8b8705ff970c8f06e consensus state has no leader
W20260810 23:32:42.130622 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 339091us
I20260810 23:32:42.157266 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 100) took 3275 ms. Trace:
I20260810 23:32:42.049561 13338 raft_consensus.cc:2454] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:42.205286 13935 raft_consensus.cc:2229] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:42.259622 13935 raft_consensus.cc:2258] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:42.220266 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046f1a7983ec40489edeaba4d2595b1c" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:42.265173 13935 ts_tablet_manager.cc:1905] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:42.175984 13514 tablet_bootstrap.cc:654] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:42.276903 12745 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:47536 (request call id 62) took 1203 ms. Trace:
I20260810 23:32:42.279127 12745 rpcz_store.cc:276] 0810 23:32:41.073789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.233765 (+159976us) service_pool.cc:227] Handling call
0810 23:32:42.206399 (+972634us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54000c21d4. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3b7268 00007f3cdf3a36cd 00007f3cddc4be04 00007f3cddc4bd92 00007f3cddc4bcc1 00007f3cddc4bc3c 00007f3cddc4aef2 00007f3cdc16c94c 00007f3cdc16bd86 00007f3cdc16b72f
0810 23:32:42.276677 (+ 70278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":293261568}
I20260810 23:32:42.280932 12237 master_service.cc:946] ReplaceTablet: received request to replace tablet 440de7d9de8d4be9a5322e1fe3388b9b from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:42.304065 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_7331598ab7f64db0bcade7763c41f59f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:42.321522 12229 master_service.cc:946] ReplaceTablet: received request to replace tablet 874ca0c3586b41a8a3074d8988813633 from {username='slave'} at 127.0.0.1:34742
W20260810 23:32:42.323103 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 169484us
W20260810 23:32:42.329766 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7331598ab7f64db0bcade7763c41f59f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:42.226198 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 187556us
I20260810 23:32:42.218194 13974 ts_tablet_manager.cc:1905] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:42.366176 12235 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6f53aca31ee4e338d9b5e281557c121 has no consensus state
I20260810 23:32:42.368134 12235 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_22eeebd76f0e44bca7287cafea0de6f7" } new_table_name: "default.Table_8a65c1c8c57e401d90d3baaa3c19af37" modify_external_catalogs: true
I20260810 23:32:42.404663 13665 tablet_bootstrap.cc:492] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:42.451493 13665 ts_tablet_manager.cc:1397] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 1.265s user 0.016s sys 0.007s
I20260810 23:32:42.425339 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b (DEFAULT_TABLE table=default.Table_d942bf134f9849718bb9eb12c24806a1 [id=ff8088bdfc9f456683dd802d62ae67a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:42.454255 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:42.222011 12958 consensus_peers.cc:487] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d596299c82b4ff1ac151d68b97ff55b. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:42.460741 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 239003us
I20260810 23:32:42.467800 13057 raft_consensus.cc:3037] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:42.474102 12958 leader_election.cc:302] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:42.290328 13691 tablet_bootstrap.cc:700] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b: Time spent opening tablet: real 0.263s user 0.004s sys 0.000s
I20260810 23:32:42.403887 14234 raft_consensus.cc:482] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:42.492712 14234 raft_consensus.cc:504] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:42.499756 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30bc9bd4e1f462b8d3be7f8a2f28f1e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:42.503352 12952 leader_election.cc:341] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:42.504662 14234 leader_election.cc:288] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:42.507222 13691 tablet_bootstrap.cc:654] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:42.513027 14121 rpcz_store.cc:276] 0810 23:32:38.900268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.903177 (+ 2909us) service_pool.cc:227] Handling call
0810 23:32:41.141599 (+2238422us) ts_tablet_manager.cc:1145] Deleting tablet dd83bbe7a1664c278098fa66b9c6895e
0810 23:32:41.141667 (+ 68us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:41.465195 (+323528us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:41.615018 (+149823us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:42.156992 (+541974us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2237548}
W20260810 23:32:42.449852 14266 log.cc:924] Time spent T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3: Append to log took a long time: real 0.287s user 0.002s sys 0.000s
I20260810 23:32:42.546305 13935 ts_tablet_manager.cc:1918] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:42.534744 14178 raft_consensus.cc:482] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:42.564767 14178 raft_consensus.cc:504] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:42.554409 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30bc9bd4e1f462b8d3be7f8a2f28f1e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:42.567322 12803 raft_consensus.cc:2454] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:42.572510 14178 leader_election.cc:288] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:42.573807 14121 tablet_replica.cc:330] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:42.574888 14121 raft_consensus.cc:2229] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:42.575573 14121 raft_consensus.cc:2258] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:42.561398 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 79204.7us
I20260810 23:32:42.580355 13935 log.cc:1196] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/c45f9dfd03764d7fb1b6fb5e28b66bd3
I20260810 23:32:42.585094 13935 ts_tablet_manager.cc:1939] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:42.600123 13514 tablet_bootstrap.cc:492] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:42.556676 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet dd83bbe7a1664c278098fa66b9c6895e (table default.Table_7c223ce60be04a498d08155325785819 [id=0eeab77fa3d54892a91f87f2fbdd673d]) successfully deleted
W20260810 23:32:42.603367 12215 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 46726.7us
I20260810 23:32:42.614563 13514 ts_tablet_manager.cc:1397] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.842s user 0.055s sys 0.105s
I20260810 23:32:42.616647 12240 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_12334169bd044f07914979e00a62582d" } new_table_name: "default.Table_d0b8422dca5e40ff919741fcee7d547c" modify_external_catalogs: true
I20260810 23:32:42.610123 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 109) took 3808 ms. Trace:
I20260810 23:32:42.616791 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_04d3b5cee61447489f042ab191392be5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:42.617314 13935 rpcz_store.cc:276] 0810 23:32:38.802249 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.845798 (+ 43549us) service_pool.cc:227] Handling call
0810 23:32:42.095720 (+3249922us) ts_tablet_manager.cc:1145] Deleting tablet c45f9dfd03764d7fb1b6fb5e28b66bd3
0810 23:32:42.095773 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:42.259198 (+163425us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54000c21d4. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3c5352 00007f3cde7db26c 00007f3cdf3e8cfa 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49 00007f3cdf3ffd11 00007f3cdf3fef3d 0000000000580784
0810 23:32:42.324749 (+ 65551us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:42.545875 (+221126us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:42.610027 (+ 64152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":245084928,"tablet-delete.queue_time_us":3249277}
I20260810 23:32:42.621882 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]) successfully deleted
I20260810 23:32:42.626607 13514 raft_consensus.cc:348] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:42.578228 12958 leader_election.cc:302] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:32:42.638782 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 60736.8us
I20260810 23:32:42.640554 14234 raft_consensus.cc:2781] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:42.640906 13514 raft_consensus.cc:374] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:42.641153 14234 raft_consensus.cc:482] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:42.625334 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 112) took 3316 ms. Trace:
I20260810 23:32:42.641633 14234 raft_consensus.cc:3037] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:42.641649 13935 rpcz_store.cc:276] 0810 23:32:39.308688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:39.308895 (+ 207us) service_pool.cc:227] Handling call
0810 23:32:42.624652 (+3315757us) ts_tablet_manager.cc:1145] Deleting tablet 13ad5d63a2104995990a86a000963cdc
0810 23:32:42.624711 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:42.625254 (+ 543us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3314896}
W20260810 23:32:42.643676 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 13ad5d63a2104995990a86a000963cdc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13ad5d63a2104995990a86a000963cdc already in progress: creating tablet
I20260810 23:32:42.644675 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30bc9bd4e1f462b8d3be7f8a2f28f1e" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:42.646497 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 116) took 3191 ms. Trace:
I20260810 23:32:42.647234 13286 tablet_service.cc:1467] Processing CreateTablet for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (DEFAULT_TABLE table=default.Table_d6f432a771c544d6acfb28ec150340da [id=2c8320255aab4036b5350d082f9e04ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:42.641292 13514 raft_consensus.cc:729] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:42.648626 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7bd9b6ca4845d8b8705ff970c8f06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:42.647919 13935 rpcz_store.cc:276] 0810 23:32:39.455207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:39.457367 (+ 2160us) service_pool.cc:227] Handling call
0810 23:32:42.645588 (+3188221us) ts_tablet_manager.cc:1145] Deleting tablet c3d4c879ca6a402dbafe91940e7b367a
0810 23:32:42.645803 (+ 215us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:42.646407 (+ 604us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3187239}
W20260810 23:32:42.653580 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet c3d4c879ca6a402dbafe91940e7b367a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3d4c879ca6a402dbafe91940e7b367a already in progress: creating tablet
I20260810 23:32:42.655062 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 119) took 2363 ms. Trace:
I20260810 23:32:42.651734 13514 consensus_queue.cc:260] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:42.671060 13273 tablet_service.cc:1467] Processing CreateTablet for tablet a41ddf9f6f71443ba75c97d2b010372c (DEFAULT_TABLE table=default.Table_73f2b1b207fb45f2adecbd37412c5e6f [id=c57f7c273a9841c1ac02523a733a383f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:42.672626 13273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41ddf9f6f71443ba75c97d2b010372c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:42.677202 13274 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (DEFAULT_TABLE table=default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:42.678635 13274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1b9d20d09c4d1ba9a4b10959807e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:42.681696 13514 ts_tablet_manager.cc:1428] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.067s user 0.007s sys 0.002s
I20260810 23:32:42.687083 13514 tablet_bootstrap.cc:492] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:42.655836 13935 rpcz_store.cc:276] 0810 23:32:40.291585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.296272 (+ 4687us) service_pool.cc:227] Handling call
0810 23:32:42.654636 (+2358364us) ts_tablet_manager.cc:1145] Deleting tablet 9c29b11192604ac18c21b7bdc0238fb3
0810 23:32:42.654692 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:42.654995 (+ 303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2355758}
I20260810 23:32:42.696324 13935 tablet_replica.cc:330] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
W20260810 23:32:42.697669 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 9c29b11192604ac18c21b7bdc0238fb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c29b11192604ac18c21b7bdc0238fb3 already in progress: creating tablet
I20260810 23:32:42.697822 13935 raft_consensus.cc:2229] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:42.698427 13935 raft_consensus.cc:2258] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:42.707536 14234 raft_consensus.cc:504] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:42.709486 13058 raft_consensus.cc:2379] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 191d120f4c7f44fc8475305a6d2e75f3 in current term 1: Already voted for candidate f26517150f1c480a877f15aec71d020b in this term.
I20260810 23:32:42.708549 13935 ts_tablet_manager.cc:1905] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:42.721215 13665 raft_consensus.cc:348] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:42.722330 13665 raft_consensus.cc:374] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:42.722673 13665 raft_consensus.cc:729] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:42.734989 14170 raft_consensus.cc:2726] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:42.738786 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30bc9bd4e1f462b8d3be7f8a2f28f1e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:42.739728 12803 raft_consensus.cc:3037] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:42.750007 13514 tablet_bootstrap.cc:654] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:42.753269 12803 raft_consensus.cc:2454] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
W20260810 23:32:42.754905 12235 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00f68708a0b74a58b69581e8324a33f6 has no consensus state
I20260810 23:32:42.723489 13665 consensus_queue.cc:260] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:42.758021 14121 ts_tablet_manager.cc:1905] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:42.720678 14266 log.cc:924] Time spent T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20260810 23:32:42.699553 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 288432us
W20260810 23:32:42.772091 13213 leader_election.cc:341] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 1f791b2df0524997ab1c53a9e5cb39ca
I20260810 23:32:42.773151 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 62705f99687340aaa1aca0942b28487f (DEFAULT_TABLE table=default.Table_96019e28037b4d38aa9fc933a29c8884 [id=dd7ca129aac0430ba386c53a30052148]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:42.776314 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04d3b5cee61447489f042ab191392be5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:42.785497 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet 124259164adf404c9cbe5f4e0d5e16c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:42.789574 13275 tablet_service.cc:1514] Processing DeleteTablet for tablet 42517f11422b42c39db282e2c5f584a3 with delete_type TABLET_DATA_DELETED (replaced by tablet a480093c481941f0bb35949c7606f37f) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:42.790668 13277 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0dd19112504f8ca57f64dcf0bd50a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:42.796290 14216 raft_consensus.cc:482] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f54aa78a60784a3bb2f5eb4bb2401ce3)
I20260810 23:32:42.797834 13691 tablet_bootstrap.cc:492] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:42.797262 14216 raft_consensus.cc:504] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:42.798421 13691 ts_tablet_manager.cc:1397] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.798s user 0.021s sys 0.000s
I20260810 23:32:42.803408 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_cc87f3e94dc84a44822a6f3af8f0c1c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:42.806946 12958 leader_election.cc:302] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:42.808589 14170 raft_consensus.cc:482] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f54aa78a60784a3bb2f5eb4bb2401ce3)
I20260810 23:32:42.809520 14236 raft_consensus.cc:2781] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:42.809608 14170 raft_consensus.cc:504] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:42.812214 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_9625301c8cda460981e80e753c785d4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:42.817441 14170 leader_election.cc:288] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:42.818176 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd110664bac34f52bfe922a3a76c6290" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:42.819701 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_3f5d5fb567214af3bcd7557741d52620"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:42.782215 14234 leader_election.cc:288] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:42.825794 14216 leader_election.cc:288] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:42.829406 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 133654008f5247b2ac87dcd09195f197 with delete_type TABLET_DATA_DELETED (replaced by tablet ca8c0d1dacc344cbacb0c5988f0e2221) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:42.827301 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c45b71432a45269cd6a106fc5f7292" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
W20260810 23:32:42.831822 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9625301c8cda460981e80e753c785d4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:42.835225 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc87f3e94dc84a44822a6f3af8f0c1c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:42.787348 13278 tablet_service.cc:1467] Processing CreateTablet for tablet 684caf5aaa8347e8b4e02323318fb0b7 (DEFAULT_TABLE table=default.Table_d4fad7cecea149e88811df34c8c11d2d [id=997d599555454bbca33e0b1d98f2f88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:42.840735 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30bc9bd4e1f462b8d3be7f8a2f28f1e" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:42.854043 13665 ts_tablet_manager.cc:1428] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.318s user 0.007s sys 0.000s
I20260810 23:32:42.858589 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c45b71432a45269cd6a106fc5f7292" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:42.861130 12952 leader_election.cc:302] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:42.862830 14216 raft_consensus.cc:2726] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:32:42.865692 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30bc9bd4e1f462b8d3be7f8a2f28f1e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:42.867348 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd110664bac34f52bfe922a3a76c6290" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:42.872869 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f5d5fb567214af3bcd7557741d52620 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:42.875602 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62705f99687340aaa1aca0942b28487f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:42.880712 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 684caf5aaa8347e8b4e02323318fb0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:42.886933 13691 raft_consensus.cc:348] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:42.888124 13691 raft_consensus.cc:374] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:42.888447 13691 raft_consensus.cc:729] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:42.889382 13691 consensus_queue.cc:260] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:42.892091 13277 tablet_service.cc:1514] Processing DeleteTablet for tablet a41ddf9f6f71443ba75c97d2b010372c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:42.897861 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet dd83bbe7a1664c278098fa66b9c6895e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:36 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:42.897656 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 46825.5us
W20260810 23:32:42.900527 12698 leader_election.cc:341] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:42.901788 12698 leader_election.cc:302] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:42.905618 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet 423e8d583d704fa9834026ebcd57e17e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:42.906903 13665 tablet_bootstrap.cc:492] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:42.915226 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 8d2f8688fc6046d58009800c5e288302 (table default.Table_783bf3bd81bb4ec294327f9ed5411afb [id=1f49f73dd9d34e968defd3a8ade21d22]) successfully deleted
I20260810 23:32:42.922034 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d596299c82b4ff1ac151d68b97ff55b" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:42.930495 12952 leader_election.cc:341] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:42.932461 12952 leader_election.cc:302] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:42.935851 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b29cda94584badb1225477ec2bd1bc" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:42.940894 14178 raft_consensus.cc:2726] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:42.970335 12214 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet a6f53aca31ee4e338d9b5e281557c121 (table default.Table_d860efaf3e3a4d1386d4c8e328463a8a [id=89c842ab64c546bbb967a2246a7220ac], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a6f53aca31ee4e338d9b5e281557c121 has no consensus state
I20260810 23:32:42.948302 13283 tablet_service.cc:1467] Processing CreateTablet for tablet a480093c481941f0bb35949c7606f37f (DEFAULT_TABLE table=default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:42.981084 14236 raft_consensus.cc:686] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:42.982383 14236 consensus_queue.cc:237] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:42.979879 13514 tablet_bootstrap.cc:492] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:42.998358 14178 raft_consensus.cc:482] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:42.999224 14178 raft_consensus.cc:504] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:43.003177 14178 leader_election.cc:288] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:43.005823 13514 ts_tablet_manager.cc:1397] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.319s user 0.020s sys 0.002s
I20260810 23:32:43.010250 13514 raft_consensus.cc:348] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:43.030414 12240 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_7ed38c2d149c496393381b8394b2ee06" } new_table_name: "default.Table_ac23e0995fcd414ead1b88406cf7d79e" modify_external_catalogs: true
I20260810 23:32:43.033887 13665 tablet_bootstrap.cc:654] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:43.036722 12234 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_8a65c1c8c57e401d90d3baaa3c19af37" } new_table_name: "default.Table_d134a55028664442b08b229593877973" modify_external_catalogs: true
I20260810 23:32:43.045133 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd77526f1db4455291fde8a001092a09" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:43.058640 13691 ts_tablet_manager.cc:1428] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.260s user 0.012s sys 0.000s
I20260810 23:32:43.061072 14216 raft_consensus.cc:482] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:43.061874 14216 raft_consensus.cc:504] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:43.064297 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd77526f1db4455291fde8a001092a09" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:43.065270 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f791b2df0524997ab1c53a9e5cb39ca" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:43.065523 13057 raft_consensus.cc:2454] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:32:43.066140 13060 raft_consensus.cc:3037] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:32:43.057905 14170 raft_consensus.cc:2726] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 23:32:43.078632 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 146331us
I20260810 23:32:43.091509 13331 raft_consensus.cc:3037] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:43.093161 13691 tablet_bootstrap.cc:492] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:43.097826 14170 consensus_queue.cc:1035] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:43.099565 14216 leader_election.cc:288] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:43.101997 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd77526f1db4455291fde8a001092a09" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:43.104135 12952 leader_election.cc:341] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:43.119431 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a480093c481941f0bb35949c7606f37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.126243 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 806fe5162aef4d36ad5e2ac324047426 (DEFAULT_TABLE table=default.Table_848da5bb390440608b71e81db171df2c [id=6920efe096de4514a4d75aa2871876b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.127573 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 806fe5162aef4d36ad5e2ac324047426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.039691 14178 raft_consensus.cc:482] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:43.164391 14178 raft_consensus.cc:504] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:43.167464 12768 tablet_service.cc:1467] Processing CreateTablet for tablet df3ac117f3b34c1bb55b6a258d4fd539 (DEFAULT_TABLE table=default.Table_20b50818a91248faac361d538df4fb30 [id=bdfbe648345f4e4e8f67c244f4a15f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.091732 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 806fe5162aef4d36ad5e2ac324047426 (DEFAULT_TABLE table=default.Table_848da5bb390440608b71e81db171df2c [id=6920efe096de4514a4d75aa2871876b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.103166 12700 leader_election.cc:302] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:43.169072 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3ac117f3b34c1bb55b6a258d4fd539. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:43.169013 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 65976.9us
I20260810 23:32:43.172336 14278 raft_consensus.cc:2781] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:43.173031 14278 raft_consensus.cc:482] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:43.173624 14278 raft_consensus.cc:3037] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:43.177325 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 806fe5162aef4d36ad5e2ac324047426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.181911 12223 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_75ca982c15e34c8d9603f100dff5b392 [id=b809029bf6cc48ab917e48bbf2109ae1]). Expected version 2 got 0
I20260810 23:32:43.182056 14178 leader_election.cc:288] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:43.184827 12223 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:43.131435 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd77526f1db4455291fde8a001092a09" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:43.157163 12763 tablet_service.cc:1514] Processing DeleteTablet for tablet b261673faa964b65af3576c89e302864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:43.161569 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 975a9bab969e46d2b939fc76ce350b99 (DEFAULT_TABLE table=default.Table_d36db682741e4ef6bde7416df55ab535 [id=da7b0a1b8cbc4acabf7fd986c43fcbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.189925 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975a9bab969e46d2b939fc76ce350b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.197572 13514 raft_consensus.cc:374] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:43.197939 13514 raft_consensus.cc:729] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:43.205066 13514 consensus_queue.cc:260] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:43.208320 13031 tablet_service.cc:1467] Processing CreateTablet for tablet 975a9bab969e46d2b939fc76ce350b99 (DEFAULT_TABLE table=default.Table_d36db682741e4ef6bde7416df55ab535 [id=da7b0a1b8cbc4acabf7fd986c43fcbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.211601 13935 ts_tablet_manager.cc:1918] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260810 23:32:43.221558 12218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.188:45477, user_credentials={real_user=slave}} blocked reactor thread for 144875us
I20260810 23:32:43.232477 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975a9bab969e46d2b939fc76ce350b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.238382 13018 tablet_service.cc:1467] Processing CreateTablet for tablet df3ac117f3b34c1bb55b6a258d4fd539 (DEFAULT_TABLE table=default.Table_20b50818a91248faac361d538df4fb30 [id=bdfbe648345f4e4e8f67c244f4a15f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.272831 12236 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 88c53785fdd24df79f138df6176e75d2 (table default.Table_d0b8422dca5e40ff919741fcee7d547c [id=71c9424f327040a68889fc257935c535]). Expected version 3 got 1
I20260810 23:32:43.286903 12236 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 2 got 0
I20260810 23:32:43.288861 12236 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:43.292603 13014 tablet_service.cc:1467] Processing CreateTablet for tablet 00f68708a0b74a58b69581e8324a33f6 (DEFAULT_TABLE table=default.Table_8a65c1c8c57e401d90d3baaa3c19af37 [id=26be70827ca442a4b24a26015ffd952a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.301560 13060 raft_consensus.cc:2454] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 2.
W20260810 23:32:43.283293 12226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 45) took 1045 ms (client timeout 1000 ms). Trace:
W20260810 23:32:43.309475 12226 rpcz_store.cc:269] 0810 23:32:42.238200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.238423 (+ 223us) service_pool.cc:227] Handling call
0810 23:32:43.283206 (+1044783us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":4}
W20260810 23:32:43.319284 12234 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00f68708a0b74a58b69581e8324a33f6 consensus state has no leader
W20260810 23:32:43.323395 12240 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd77526f1db4455291fde8a001092a09 consensus state has no leader
I20260810 23:32:43.328986 12237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 440de7d9de8d4be9a5322e1fe3388b9b
I20260810 23:32:43.339885 12237 catalog_manager.cc:6565] ReplaceTablet: tablet 440de7d9de8d4be9a5322e1fe3388b9b deleted and replaced by tablet e62dc27b59ba469d99e8d11372aceff5
I20260810 23:32:43.340333 12237 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 212) took 1060 ms. Trace:
I20260810 23:32:43.340975 12237 rpcz_store.cc:276] 0810 23:32:42.280403 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.280600 (+ 197us) service_pool.cc:227] Handling call
0810 23:32:43.340263 (+1059663us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:43.260488 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:43.260882 (+ 394us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:43.260906 (+ 24us) write_op.cc:435] Acquired schema lock
0810 23:32:43.260919 (+ 13us) tablet.cc:660] Decoding operations
0810 23:32:43.261283 (+ 364us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:43.261342 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:43.261365 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:43.261584 (+ 219us) tablet.cc:699] Row locks acquired
0810 23:32:43.261600 (+ 16us) write_op.cc:260] PREPARE: finished
0810 23:32:43.261723 (+ 123us) write_op.cc:270] Start()
0810 23:32:43.264483 (+ 2760us) write_op.cc:276] Timestamp: P: 1786404763263911 usec, L: 0
0810 23:32:43.264509 (+ 26us) op_driver.cc:331] REPLICATION: starting
0810 23:32:43.265091 (+ 582us) log.cc:842] Serialized 631 byte log entry
0810 23:32:43.324856 (+ 59765us) op_driver.cc:444] REPLICATION: finished
0810 23:32:43.326055 (+ 1199us) write_op.cc:301] APPLY: starting
0810 23:32:43.326152 (+ 97us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:43.326331 (+ 179us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:43.326446 (+ 115us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:43.326946 (+ 500us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:43.327319 (+ 373us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:43.327424 (+ 105us) write_op.cc:312] APPLY: finished
0810 23:32:43.327785 (+ 361us) log.cc:842] Serialized 29 byte log entry
0810 23:32:43.327990 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:43.328114 (+ 124us) write_op.cc:454] Released schema lock
0810 23:32:43.328186 (+ 72us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:43.329640 (+ 1454us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:43.329699 (+ 59us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":955,"apply.run_cpu_time_us":2847,"apply.run_wall_time_us":3942,"num_ops":2,"prepare.queue_time_us":103105,"prepare.run_cpu_time_us":2195,"prepare.run_wall_time_us":4816,"replication_time_us":60233}]]}
I20260810 23:32:43.370223 12228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 4f3049bcdc6e43c795961c74f0d008f5
I20260810 23:32:43.375267 12228 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 208) took 1497 ms. Trace:
I20260810 23:32:43.375765 12228 rpcz_store.cc:276] 0810 23:32:41.874462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.874655 (+ 193us) service_pool.cc:227] Handling call
0810 23:32:41.881219 (+ 6564us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:41.881947 (+ 728us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:41.882134 (+ 187us) catalog_manager.cc:2761] Locking tablets
0810 23:32:43.059257 (+1177123us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:43.369674 (+310417us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:43.369981 (+ 307us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:43.370109 (+ 128us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:43.371996 (+ 1887us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:43.265443 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:43.265836 (+ 393us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:43.265860 (+ 24us) write_op.cc:435] Acquired schema lock
0810 23:32:43.265872 (+ 12us) tablet.cc:660] Decoding operations
0810 23:32:43.266213 (+ 341us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:43.266276 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:43.266315 (+ 39us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:43.266783 (+ 468us) tablet.cc:699] Row locks acquired
0810 23:32:43.266805 (+ 22us) write_op.cc:260] PREPARE: finished
0810 23:32:43.266954 (+ 149us) write_op.cc:270] Start()
0810 23:32:43.267056 (+ 102us) write_op.cc:276] Timestamp: P: 1786404763269139 usec, L: 0
0810 23:32:43.267073 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:32:43.267613 (+ 540us) log.cc:842] Serialized 734 byte log entry
0810 23:32:43.325272 (+ 57659us) op_driver.cc:444] REPLICATION: finished
0810 23:32:43.330252 (+ 4980us) write_op.cc:301] APPLY: starting
0810 23:32:43.330367 (+ 115us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:43.330554 (+ 187us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:43.330657 (+ 103us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:43.331051 (+ 394us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:43.331316 (+ 265us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:43.331397 (+ 81us) write_op.cc:312] APPLY: finished
0810 23:32:43.331737 (+ 340us) log.cc:842] Serialized 29 byte log entry
0810 23:32:43.331949 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:43.364766 (+ 32817us) write_op.cc:454] Released schema lock
0810 23:32:43.364840 (+ 74us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4784,"num_ops":2,"prepare.queue_time_us":107568,"prepare.run_cpu_time_us":2379,"prepare.run_wall_time_us":2377,"replication_time_us":58069,"threads_started":1}]]}
W20260810 23:32:43.408485 12223 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 28) took 1592 ms (client timeout 1000 ms). Trace:
W20260810 23:32:43.410128 12223 rpcz_store.cc:269] 0810 23:32:41.816239 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.816497 (+ 258us) service_pool.cc:227] Handling call
0810 23:32:43.408366 (+1591869us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18328,"reported_tablets":10}
I20260810 23:32:43.431473 13665 tablet_bootstrap.cc:492] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:43.441286 13974 ts_tablet_manager.cc:1918] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:43.442814 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_69797cc483424ae99a99731b36130f0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:43.445122 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_a7072962772044ad8f3e9a8e446baea1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:43.446223 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_65718901959a4edaaa7370e442209578"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:43.446843 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7072962772044ad8f3e9a8e446baea1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:43.448210 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65718901959a4edaaa7370e442209578 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:43.445135 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69797cc483424ae99a99731b36130f0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:43.306648 13691 tablet_bootstrap.cc:654] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:43.466658 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 27c39d751e544d9a8afa26dc935a819a (DEFAULT_TABLE table=default.Table_78e8fac0b43b408abf61639343c2e8e8 [id=f8efd2f66f3c4968932a2cf4f70a7154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.310117 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3a834d77474c3d8feaa5a01f3faff3" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:43.316510 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3ac117f3b34c1bb55b6a258d4fd539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.324697 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ba75f2c2b54ec1968db8feeacf8383 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c29b11192604ac18c21b7bdc0238fb3) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:43.325424 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet 840400119d20424ea0f03f2bff550d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:43.347960 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet 440de7d9de8d4be9a5322e1fe3388b9b with delete_type TABLET_DATA_DELETED (replaced by tablet e62dc27b59ba469d99e8d11372aceff5) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:43.376621 14216 consensus_queue.cc:1035] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:43.378387 14236 consensus_queue.cc:1035] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:43.382808 13007 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3049bcdc6e43c795961c74f0d008f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:43.389401 13020 tablet_service.cc:1467] Processing CreateTablet for tablet f1452c292e9c476b9e26f56998f0ad55 (DEFAULT_TABLE table=default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.376621 14121 ts_tablet_manager.cc:1918] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:43.477303 14121 log.cc:1196] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/8d2f8688fc6046d58009800c5e288302
I20260810 23:32:43.478588 14121 ts_tablet_manager.cc:1939] T 8d2f8688fc6046d58009800c5e288302 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:43.416060 13008 tablet_service.cc:1467] Processing CreateTablet for tablet ca8c0d1dacc344cbacb0c5988f0e2221 (DEFAULT_TABLE table=default.Table_d664d07d116e4f2aa4fc263449eb8dd0 [id=04ed6428e4df49b2ab0438035da70731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.473978 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c39d751e544d9a8afa26dc935a819a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.484427 13007 tablet_service.cc:1467] Processing CreateTablet for tablet afc0e5cb08c44f7b9662d63905d99b02 (DEFAULT_TABLE table=default.Table_d239f0c3798a49c1bf1055dab20c108a [id=845a7c2ecd844114a10dd1f7fe03ba7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.485611 13008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8c0d1dacc344cbacb0c5988f0e2221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.486819 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1452c292e9c476b9e26f56998f0ad55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.493593 13007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc0e5cb08c44f7b9662d63905d99b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.500620 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 101) took 4599 ms. Trace:
I20260810 23:32:43.505903 13665 ts_tablet_manager.cc:1397] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.606s user 0.009s sys 0.024s
I20260810 23:32:43.510813 14121 rpcz_store.cc:276] 0810 23:32:38.901004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.904253 (+ 3249us) service_pool.cc:227] Handling call
0810 23:32:42.515768 (+3611515us) ts_tablet_manager.cc:1145] Deleting tablet 8d2f8688fc6046d58009800c5e288302
0810 23:32:42.515887 (+ 119us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:43.180475 (+664588us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:43.376138 (+195663us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:43.500488 (+124350us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3609709}
I20260810 23:32:43.513774 14121 tablet_replica.cc:330] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:43.514420 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 8d2f8688fc6046d58009800c5e288302 (table default.Table_783bf3bd81bb4ec294327f9ed5411afb [id=1f49f73dd9d34e968defd3a8ade21d22]) successfully deleted
I20260810 23:32:43.517590 14121 raft_consensus.cc:2229] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:43.522593 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_90a489ddbc1e4f7bb710756a1c40465f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:43.537424 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90a489ddbc1e4f7bb710756a1c40465f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:43.531977 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00f68708a0b74a58b69581e8324a33f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.554556 12235 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:43.567000 12228 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_529a79b85b6f4c5da1ab44678a564943" } new_table_name: "default.Table_3b251f8589be43f29704b8906d49f6cd" modify_external_catalogs: true
I20260810 23:32:43.571774 12237 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_34eb8fd429274b5d8af145c70aab107b" } new_table_name: "default.Table_f577b98edfc0492cb8edbc6198d6bcd1" modify_external_catalogs: true
I20260810 23:32:43.581607 12235 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 6a3a834d77474c3d8feaa5a01f3faff3 (table default.Table_dfb60e6e40874a5da0c56f6926ce1f63 [id=1ea3dfab929845f89d4680129ae45d94]). Expected version 1 got 0
I20260810 23:32:43.586974 12229 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 214) took 1266 ms. Trace:
I20260810 23:32:43.587613 12229 rpcz_store.cc:276] 0810 23:32:42.321144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.321325 (+ 181us) service_pool.cc:227] Handling call
0810 23:32:43.586879 (+1265554us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:43.572876 14121 raft_consensus.cc:2258] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:43.599097 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_75ca982c15e34c8d9603f100dff5b392 [id=b809029bf6cc48ab917e48bbf2109ae1]). Expected version 2 got 0
I20260810 23:32:43.600485 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:43.601210 13691 tablet_bootstrap.cc:492] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:43.601593 12224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 874ca0c3586b41a8a3074d8988813633
I20260810 23:32:43.603307 12224 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 206) took 1864 ms. Trace:
I20260810 23:32:43.604604 13691 ts_tablet_manager.cc:1397] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.520s user 0.025s sys 0.000s
I20260810 23:32:43.618937 12229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_26bf2c7d86704b5b981ecebb4e90ba75" } modify_external_catalogs: true
I20260810 23:32:43.619957 12229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_26bf2c7d86704b5b981ecebb4e90ba75" } modify_external_catalogs: true
I20260810 23:32:43.603636 12224 rpcz_store.cc:276] 0810 23:32:41.738701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.739043 (+ 342us) service_pool.cc:227] Handling call
0810 23:32:41.821131 (+ 82088us) spinlock_profiling.cc:243] Waited 80.8 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7466 00007f0ce6e28b34 00007f0ce6e28ac2 00007f0ce6e289f1 00007f0ce6e2896c 00007f0ce6e27c22 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:41.830951 (+ 9820us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:41.831810 (+ 859us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:41.833780 (+ 1970us) catalog_manager.cc:2761] Locking tablets
0810 23:32:43.129327 (+1295547us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:43.496180 (+366853us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:43.545498 (+ 49318us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:43.601483 (+ 55985us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:43.603201 (+ 1718us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:43.267933 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:43.339744 (+ 71811us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:43.339812 (+ 68us) write_op.cc:435] Acquired schema lock
0810 23:32:43.339827 (+ 15us) tablet.cc:660] Decoding operations
0810 23:32:43.341078 (+ 1251us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:43.341164 (+ 86us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:43.341230 (+ 66us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:43.380458 (+ 39228us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b0c0009ddd0. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce76ac361 00007f0ce77e6804 00007f0ce77e710d 00007f0ce77e700e 00007f0ce778b7c5 00007f0ce768affd 00007f0ce7788de5 00007f0ce7774603 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59
0810 23:32:43.380520 (+ 62us) tablet.cc:699] Row locks acquired
0810 23:32:43.380535 (+ 15us) write_op.cc:260] PREPARE: finished
0810 23:32:43.380705 (+ 170us) write_op.cc:270] Start()
0810 23:32:43.380997 (+ 292us) write_op.cc:276] Timestamp: P: 1786404763382872 usec, L: 0
0810 23:32:43.381016 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 23:32:43.440785 (+ 59769us) log.cc:842] Serialized 765 byte log entry
0810 23:32:43.452536 (+ 11751us) spinlock_profiling.cc:243] Waited 58.8 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:43.492050 (+ 39514us) op_driver.cc:444] REPLICATION: finished
0810 23:32:43.492581 (+ 531us) write_op.cc:301] APPLY: starting
0810 23:32:43.492673 (+ 92us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:43.492918 (+ 245us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:43.492983 (+ 65us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:43.493458 (+ 475us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:43.493703 (+ 245us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:43.493795 (+ 92us) write_op.cc:312] APPLY: finished
0810 23:32:43.494256 (+ 461us) log.cc:842] Serialized 29 byte log entry
0810 23:32:43.494627 (+ 371us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:43.494778 (+ 151us) write_op.cc:454] Released schema lock
0810 23:32:43.494839 (+ 61us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:43.495222 (+ 383us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:43.495289 (+ 67us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":185760512,"child_traces":[["op",{"apply.queue_time_us":154,"apply.run_cpu_time_us":3100,"apply.run_wall_time_us":3099,"num_ops":2,"prepare.queue_time_us":74898,"prepare.run_cpu_time_us":7737,"prepare.run_wall_time_us":194395,"raft.queue_time_us":119,"raft.run_cpu_time_us":1246,"raft.run_wall_time_us":4851,"replication_time_us":110861,"spinlock_wait_cycles":194682496}]]}
I20260810 23:32:43.512976 13212 leader_election.cc:302] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
W20260810 23:32:43.663022 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 150168us
I20260810 23:32:43.664655 13212 leader_election.cc:302] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
W20260810 23:32:43.214654 12698 leader_election.cc:341] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:43.684525 13935 log.cc:1196] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/a0ba75f2c2b54ec1968db8feeacf8383
I20260810 23:32:43.686136 13935 ts_tablet_manager.cc:1939] T a0ba75f2c2b54ec1968db8feeacf8383 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:43.218932 12763 tablet_service.cc:1467] Processing CreateTablet for tablet 00f68708a0b74a58b69581e8324a33f6 (DEFAULT_TABLE table=default.Table_8a65c1c8c57e401d90d3baaa3c19af37 [id=26be70827ca442a4b24a26015ffd952a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.241504 14178 raft_consensus.cc:482] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:32:43.280210 14317 log.cc:924] Time spent T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3: Append to log took a long time: real 0.275s user 0.001s sys 0.000s
I20260810 23:32:43.381031 12767 tablet_service.cc:1514] Processing DeleteTablet for tablet 440de7d9de8d4be9a5322e1fe3388b9b with delete_type TABLET_DATA_DELETED (replaced by tablet e62dc27b59ba469d99e8d11372aceff5) from {username='slave'} at 127.0.0.1:47546
W20260810 23:32:43.475521 12700 leader_election.cc:341] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:43.381635 12760 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3049bcdc6e43c795961c74f0d008f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:43.478931 12761 tablet_service.cc:1467] Processing CreateTablet for tablet afc0e5cb08c44f7b9662d63905d99b02 (DEFAULT_TABLE table=default.Table_d239f0c3798a49c1bf1055dab20c108a [id=845a7c2ecd844114a10dd1f7fe03ba7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.383591 12764 tablet_service.cc:1467] Processing CreateTablet for tablet f1452c292e9c476b9e26f56998f0ad55 (DEFAULT_TABLE table=default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.284969 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d1b9d20d09c4d1ba9a4b10959807e50) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:43.384656 12769 tablet_service.cc:1467] Processing CreateTablet for tablet ca8c0d1dacc344cbacb0c5988f0e2221 (DEFAULT_TABLE table=default.Table_d664d07d116e4f2aa4fc263449eb8dd0 [id=04ed6428e4df49b2ab0438035da70731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.446839 12766 tablet_service.cc:1467] Processing CreateTablet for tablet 27c39d751e544d9a8afa26dc935a819a (DEFAULT_TABLE table=default.Table_78e8fac0b43b408abf61639343c2e8e8 [id=f8efd2f66f3c4968932a2cf4f70a7154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:43.252827 13514 ts_tablet_manager.cc:1428] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.247s user 0.008s sys 0.000s
I20260810 23:32:43.503857 14278 raft_consensus.cc:504] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:43.608600 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet 874ca0c3586b41a8a3074d8988813633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:43.687917 14178 raft_consensus.cc:504] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:43.698477 14178 leader_election.cc:288] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
W20260810 23:32:43.688439 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 212995us
I20260810 23:32:43.703747 12235 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:43.670763 14206 raft_consensus.cc:2726] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:43.705421 14206 raft_consensus.cc:2726] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:43.693619 14278 leader_election.cc:288] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:43.694801 12805 raft_consensus.cc:2379] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate 191d120f4c7f44fc8475305a6d2e75f3 in this term.
W20260810 23:32:43.694937 13514 ts_tablet_manager.cc:1460] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3: Tablet startup took 688ms
W20260810 23:32:43.712016 13514 ts_tablet_manager.cc:1462] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3: Trace:
0810 23:32:35.990960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:35.991664 (+ 704us) service_pool.cc:227] Handling call
0810 23:32:35.993481 (+ 1817us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:35.993577 (+ 96us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:36.185394 (+191817us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:36.365834 (+180440us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:36.366394 (+ 560us) inbound_call.cc:173] Queueing success response
0810 23:32:42.686640 (+6320246us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0810 23:32:43.006300 (+319660us) ts_tablet_manager.cc:1429] Starting tablet replica
0810 23:32:43.007924 (+ 1624us) tablet_replica.cc:259] Starting instrumentation
0810 23:32:43.008696 (+ 772us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":32593,"raft.run_cpu_time_us":500,"raft.run_wall_time_us":2572,"tablet-open.queue_time_us":6320353}
I20260810 23:32:43.715745 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00f68708a0b74a58b69581e8324a33f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.717360 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 120) took 3078 ms. Trace:
I20260810 23:32:43.717744 13935 rpcz_store.cc:276] 0810 23:32:40.697903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.698059 (+ 156us) service_pool.cc:227] Handling call
0810 23:32:42.695992 (+1997933us) ts_tablet_manager.cc:1145] Deleting tablet a0ba75f2c2b54ec1968db8feeacf8383
0810 23:32:42.696079 (+ 87us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:42.707485 (+ 11406us) spinlock_profiling.cc:243] Waited 6.25 ms on lock 0x7b280003d108. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3ce02ae8b1 00007f3ce02c21ea 00007f3cde72e49f 00007f3cde7db3fb 00007f3cdf3e8cfa 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49 00007f3cdf3ffd11 00007f3cdf3fef3d
0810 23:32:42.967604 (+260119us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:43.211317 (+243713us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:43.717288 (+505971us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14386432,"tablet-delete.queue_time_us":1997165}
I20260810 23:32:43.720207 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 121) took 3020 ms. Trace:
I20260810 23:32:43.720566 13935 rpcz_store.cc:276] 0810 23:32:40.699673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.699921 (+ 248us) service_pool.cc:227] Handling call
0810 23:32:43.719899 (+3019978us) ts_tablet_manager.cc:1145] Deleting tablet e5b08ed744b3420f98d241ad96bca6d0
0810 23:32:43.719941 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:43.720147 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3018818}
I20260810 23:32:43.723666 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1452c292e9c476b9e26f56998f0ad55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.725062 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet a0ba75f2c2b54ec1968db8feeacf8383 (table default.Table_dd2cd30e5e2349de9180be01866dc513 [id=8754da7be72b4eef80daacdcdfde82a7]) successfully deleted
I20260810 23:32:43.728571 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc0e5cb08c44f7b9662d63905d99b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.730057 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8c0d1dacc344cbacb0c5988f0e2221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.732641 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c39d751e544d9a8afa26dc935a819a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:43.733995 14278 raft_consensus.cc:482] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:43.734639 14278 raft_consensus.cc:504] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:43.737465 14278 leader_election.cc:288] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
W20260810 23:32:43.752861 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet e5b08ed744b3420f98d241ad96bca6d0 because the tablet was not found. No further retry: Not found: Tablet not found: e5b08ed744b3420f98d241ad96bca6d0
I20260810 23:32:43.749069 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 122) took 3047 ms. Trace:
I20260810 23:32:43.756455 13935 rpcz_store.cc:276] 0810 23:32:40.702076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.702191 (+ 115us) service_pool.cc:227] Handling call
0810 23:32:43.748677 (+3046486us) ts_tablet_manager.cc:1145] Deleting tablet 1f791b2df0524997ab1c53a9e5cb39ca
0810 23:32:43.748736 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:43.748981 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3045835}
I20260810 23:32:43.757481 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_764f15b83d234a90886ebfae787847ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:43.759662 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_764f15b83d234a90886ebfae787847ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:43.766569 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 1f791b2df0524997ab1c53a9e5cb39ca because the tablet was not found. No further retry: Not found: Tablet not found: 1f791b2df0524997ab1c53a9e5cb39ca
I20260810 23:32:43.767146 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 123) took 3063 ms. Trace:
I20260810 23:32:43.767514 13935 rpcz_store.cc:276] 0810 23:32:40.703682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.703815 (+ 133us) service_pool.cc:227] Handling call
0810 23:32:43.766846 (+3063031us) ts_tablet_manager.cc:1145] Deleting tablet 840400119d20424ea0f03f2bff550d44
0810 23:32:43.766891 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:43.767079 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3062501}
W20260810 23:32:43.770816 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 840400119d20424ea0f03f2bff550d44 because the tablet was not found. No further retry: Not found: Tablet not found: 840400119d20424ea0f03f2bff550d44
I20260810 23:32:43.771283 13935 tablet_replica.cc:330] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:43.772112 13935 raft_consensus.cc:2229] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:43.772682 13935 raft_consensus.cc:2258] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:43.687378 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 528308us
I20260810 23:32:43.793087 13935 ts_tablet_manager.cc:1905] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:43.627003 14341 log.cc:924] Time spent T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.090s user 0.002s sys 0.000s
I20260810 23:32:43.442634 13974 log.cc:1196] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/1f791b2df0524997ab1c53a9e5cb39ca
I20260810 23:32:43.831264 13974 ts_tablet_manager.cc:1939] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:43.831626 13665 raft_consensus.cc:348] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:43.832448 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 111) took 5160 ms. Trace:
I20260810 23:32:43.832829 13974 rpcz_store.cc:276] 0810 23:32:38.671761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.882712 (+210951us) service_pool.cc:227] Handling call
0810 23:32:41.214656 (+2331944us) ts_tablet_manager.cc:1145] Deleting tablet 1f791b2df0524997ab1c53a9e5cb39ca
0810 23:32:41.214759 (+ 103us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:41.978929 (+764170us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b5400100254. stack: 00007f8764d74033 00007f8764d73f30 00007f87648f5534 00000000004db2b4 00000000004db249 00007f876bc80471 00007f876bd03352 00007f876c11926c 00007f876cd26cfa 00007f876cd56e10 00007f876cd3de0d 00007f876cd3ddb9 00007f876cd3dd49 00007f876cd3dd11 00007f876cd3cf3d 0000000000580784
0810 23:32:43.053990 (+1075061us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:43.440681 (+386691us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:43.832339 (+391658us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":160971,"spinlock_wait_cycles":971817216,"tablet-delete.queue_time_us":2317639}
I20260810 23:32:43.833858 13665 raft_consensus.cc:374] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:43.834342 13665 raft_consensus.cc:729] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:43.834762 13974 tablet_replica.cc:330] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:43.834964 12224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_848da5bb390440608b71e81db171df2c" } modify_external_catalogs: true
I20260810 23:32:43.835148 13665 consensus_queue.cc:260] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:43.827361 13276 consensus_meta.cc:287] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent flushing consensus metadata: real 0.508s user 0.005s sys 0.007s
I20260810 23:32:43.835913 12224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_848da5bb390440608b71e81db171df2c" } modify_external_catalogs: true
W20260810 23:32:43.806360 13338 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.562s user 0.014s sys 0.022s
I20260810 23:32:43.838603 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet 874ca0c3586b41a8a3074d8988813633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:43.842574 13276 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 134) took 1420 ms. Trace:
I20260810 23:32:43.843145 13276 rpcz_store.cc:276] 0810 23:32:42.422120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.422312 (+ 192us) service_pool.cc:227] Handling call
0810 23:32:42.453136 (+ 30824us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:42.453353 (+ 217us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:43.157430 (+704077us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:43.841725 (+684295us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:43.842361 (+ 636us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:43.825608 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124259164adf404c9cbe5f4e0d5e16c0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:43.838977 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42517f11422b42c39db282e2c5f584a3" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:43.863797 13691 raft_consensus.cc:348] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:43.878535 13691 raft_consensus.cc:374] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:43.888062 13935 ts_tablet_manager.cc:1918] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:43.891481 13935 log.cc:1196] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/fd110664bac34f52bfe922a3a76c6290
I20260810 23:32:43.893622 13935 ts_tablet_manager.cc:1939] T fd110664bac34f52bfe922a3a76c6290 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
W20260810 23:32:43.899214 12700 leader_election.cc:341] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:43.908550 13691 raft_consensus.cc:729] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:43.909575 13691 consensus_queue.cc:260] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:43.911888 13059 raft_consensus.cc:2454] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:32:43.863797 13278 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 139) took 1083 ms. Trace:
I20260810 23:32:43.915484 13278 rpcz_store.cc:276] 0810 23:32:42.780377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.784415 (+ 4038us) service_pool.cc:227] Handling call
0810 23:32:42.875737 (+ 91322us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:42.876222 (+ 485us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:43.154682 (+278460us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:43.788311 (+633629us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:43.863700 (+ 75389us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:43.901670 12958 leader_election.cc:302] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:32:43.888430 12952 consensus_peers.cc:487] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:43.936918 14319 raft_consensus.cc:3037] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:43.948827 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd77526f1db4455291fde8a001092a09" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:43.965262 14346 tablet.cc:1722] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:43.977921 13974 raft_consensus.cc:2229] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:43.978768 13974 raft_consensus.cc:2258] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:43.981487 13974 ts_tablet_manager.cc:1905] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:43.986538 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 159847us
I20260810 23:32:43.983560 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 124) took 3278 ms. Trace:
I20260810 23:32:43.988857 13935 rpcz_store.cc:276] 0810 23:32:40.705053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.758732 (+ 53679us) service_pool.cc:227] Handling call
0810 23:32:43.771107 (+3012375us) ts_tablet_manager.cc:1145] Deleting tablet fd110664bac34f52bfe922a3a76c6290
0810 23:32:43.771152 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:43.823555 (+ 52403us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:43.879694 (+ 56139us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:43.983449 (+103755us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3011660}
I20260810 23:32:43.990290 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 125) took 3230 ms. Trace:
I20260810 23:32:43.990629 13935 rpcz_store.cc:276] 0810 23:32:40.759922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.760060 (+ 138us) service_pool.cc:227] Handling call
0810 23:32:43.989967 (+3229907us) ts_tablet_manager.cc:1145] Deleting tablet 81e26fe4d42445279f12bb17b2841d08
0810 23:32:43.990015 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:43.990224 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3229322}
I20260810 23:32:43.992898 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet fd110664bac34f52bfe922a3a76c6290 (table default.Table_0af7ca97e51c47a29d5dd8867b152066 [id=32144cd75df84cfabce64657802c111a]) successfully deleted
W20260810 23:32:43.993656 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 81e26fe4d42445279f12bb17b2841d08 because the tablet was not found. No further retry: Not found: Tablet not found: 81e26fe4d42445279f12bb17b2841d08
I20260810 23:32:43.991775 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 126) took 3229 ms. Trace:
I20260810 23:32:43.994429 13935 rpcz_store.cc:276] 0810 23:32:40.761991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.762111 (+ 120us) service_pool.cc:227] Handling call
0810 23:32:43.991506 (+3229395us) ts_tablet_manager.cc:1145] Deleting tablet 79da9583d3524347a9be09e35affaaad
0810 23:32:43.991550 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:43.991717 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3228807}
I20260810 23:32:43.996062 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 127) took 3219 ms. Trace:
W20260810 23:32:43.996742 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 79da9583d3524347a9be09e35affaaad because the tablet was not found. No further retry: Not found: Tablet not found: 79da9583d3524347a9be09e35affaaad
I20260810 23:32:43.997534 13935 rpcz_store.cc:276] 0810 23:32:40.776685 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.776833 (+ 148us) service_pool.cc:227] Handling call
0810 23:32:43.995730 (+3218897us) ts_tablet_manager.cc:1145] Deleting tablet 0c0d6cd2ad1d4a65929b0d856a171592
0810 23:32:43.995783 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:43.995973 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3218011}
W20260810 23:32:43.999508 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 0c0d6cd2ad1d4a65929b0d856a171592 because the tablet was not found. No further retry: Not found: Tablet not found: 0c0d6cd2ad1d4a65929b0d856a171592
W20260810 23:32:44.001590 12214 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 00f68708a0b74a58b69581e8324a33f6 (table default.Table_d134a55028664442b08b229593877973 [id=26be70827ca442a4b24a26015ffd952a], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 00f68708a0b74a58b69581e8324a33f6 consensus state has no leader
I20260810 23:32:44.001922 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3a834d77474c3d8feaa5a01f3faff3" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:43.974012 13514 tablet_bootstrap.cc:492] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:32:44.010586 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 346037us
I20260810 23:32:44.024912 13212 leader_election.cc:302] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
W20260810 23:32:44.039849 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 138326us
I20260810 23:32:44.030103 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd77526f1db4455291fde8a001092a09" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:44.045337 13691 ts_tablet_manager.cc:1428] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.440s user 0.008s sys 0.000s
W20260810 23:32:44.068799 13212 raft_consensus.cc:2670] T 1f791b2df0524997ab1c53a9e5cb39ca P f54aa78a60784a3bb2f5eb4bb2401ce3: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260810 23:32:44.069530 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 44799.7us
I20260810 23:32:44.072640 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 1f791b2df0524997ab1c53a9e5cb39ca (table default.Table_089a1834ee404319b9d766443f10704f [id=a74d0ca6962e44c5801d194da36bf261]) successfully deleted
I20260810 23:32:44.084368 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 129) took 3298 ms. Trace:
I20260810 23:32:44.084987 13935 rpcz_store.cc:276] 0810 23:32:40.786365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.792463 (+ 6098us) service_pool.cc:227] Handling call
0810 23:32:44.000615 (+3208152us) ts_tablet_manager.cc:1145] Deleting tablet 8b1f7c16b6bc4dc0bac80ac55332f056
0810 23:32:44.000679 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.084282 (+ 83603us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3207444}
W20260810 23:32:44.086745 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 because the tablet was not found. No further retry: Not found: Tablet not found: 8b1f7c16b6bc4dc0bac80ac55332f056
I20260810 23:32:44.089144 13935 tablet_replica.cc:330] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:44.090085 13935 raft_consensus.cc:2229] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:44.090747 13935 raft_consensus.cc:2258] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:44.099066 14368 tablet.cc:1722] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:44.067030 14121 ts_tablet_manager.cc:1905] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:44.113337 13691 ts_tablet_manager.cc:1460] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b: Tablet startup took 507ms
I20260810 23:32:44.112969 13278 tablet_service.cc:1467] Processing CreateTablet for tablet 806fe5162aef4d36ad5e2ac324047426 (DEFAULT_TABLE table=default.Table_848da5bb390440608b71e81db171df2c [id=6920efe096de4514a4d75aa2871876b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:44.115097 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 806fe5162aef4d36ad5e2ac324047426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:44.117789 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 975a9bab969e46d2b939fc76ce350b99 (DEFAULT_TABLE table=default.Table_d36db682741e4ef6bde7416df55ab535 [id=da7b0a1b8cbc4acabf7fd986c43fcbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:44.120007 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975a9bab969e46d2b939fc76ce350b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:44.120842 13274 tablet_service.cc:1467] Processing CreateTablet for tablet df3ac117f3b34c1bb55b6a258d4fd539 (DEFAULT_TABLE table=default.Table_20b50818a91248faac361d538df4fb30 [id=bdfbe648345f4e4e8f67c244f4a15f17]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:44.104012 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 93882.8us
I20260810 23:32:44.122957 13274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3ac117f3b34c1bb55b6a258d4fd539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:44.125675 13273 tablet_service.cc:1467] Processing CreateTablet for tablet 00f68708a0b74a58b69581e8324a33f6 (DEFAULT_TABLE table=default.Table_8a65c1c8c57e401d90d3baaa3c19af37 [id=26be70827ca442a4b24a26015ffd952a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:44.133603 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124259164adf404c9cbe5f4e0d5e16c0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:44.135401 13273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00f68708a0b74a58b69581e8324a33f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:44.136982 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42517f11422b42c39db282e2c5f584a3" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:44.142220 13283 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 145) took 1305 ms. Trace:
I20260810 23:32:44.160578 14376 tablet.cc:1722] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260810 23:32:44.164613 13018 consensus_meta.cc:287] T df3ac117f3b34c1bb55b6a258d4fd539 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.530s user 0.015s sys 0.000s
I20260810 23:32:44.150722 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 440de7d9de8d4be9a5322e1fe3388b9b with delete_type TABLET_DATA_DELETED (replaced by tablet e62dc27b59ba469d99e8d11372aceff5) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:44.154400 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3049bcdc6e43c795961c74f0d008f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:44.166622 13291 tablet_service.cc:1467] Processing CreateTablet for tablet f1452c292e9c476b9e26f56998f0ad55 (DEFAULT_TABLE table=default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:44.173827 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1452c292e9c476b9e26f56998f0ad55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:44.173522 13282 tablet_service.cc:1467] Processing CreateTablet for tablet ca8c0d1dacc344cbacb0c5988f0e2221 (DEFAULT_TABLE table=default.Table_d664d07d116e4f2aa4fc263449eb8dd0 [id=04ed6428e4df49b2ab0438035da70731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:44.171422 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ba75f2c2b54ec1968db8feeacf8383 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c29b11192604ac18c21b7bdc0238fb3) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:44.175925 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8c0d1dacc344cbacb0c5988f0e2221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:44.173069 13283 rpcz_store.cc:276] 0810 23:32:42.836891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.946536 (+109645us) service_pool.cc:227] Handling call
0810 23:32:42.980838 (+ 34302us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:43.033001 (+ 52163us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:43.826315 (+793314us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.140963 (+314648us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:44.142095 (+ 1132us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 23:32:44.172106 12952 consensus_peers.cc:487] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:44.184566 13277 tablet_service.cc:1467] Processing CreateTablet for tablet 27c39d751e544d9a8afa26dc935a819a (DEFAULT_TABLE table=default.Table_78e8fac0b43b408abf61639343c2e8e8 [id=f8efd2f66f3c4968932a2cf4f70a7154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:44.188225 13277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c39d751e544d9a8afa26dc935a819a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:44.079815 12700 leader_election.cc:302] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:44.196199 14343 raft_consensus.cc:2781] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:44.197266 14343 raft_consensus.cc:482] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:44.198065 14343 raft_consensus.cc:3037] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:44.213893 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_98eeae0f1e684e5393147f22149b771a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:44.216759 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98eeae0f1e684e5393147f22149b771a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:44.217147 13935 ts_tablet_manager.cc:1905] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:44.220191 14319 raft_consensus.cc:2726] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:44.203933 13283 tablet_service.cc:1467] Processing CreateTablet for tablet afc0e5cb08c44f7b9662d63905d99b02 (DEFAULT_TABLE table=default.Table_d239f0c3798a49c1bf1055dab20c108a [id=845a7c2ecd844114a10dd1f7fe03ba7e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:44.224470 12698 leader_election.cc:341] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:44.233769 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_77274019ccb14c57b2a9ce882ee84dac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:44.239027 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77274019ccb14c57b2a9ce882ee84dac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:44.237432 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b29cda94584badb1225477ec2bd1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:33 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:44.238233 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 874ca0c3586b41a8a3074d8988813633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:44.240661 13665 ts_tablet_manager.cc:1428] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.724s user 0.014s sys 0.016s
I20260810 23:32:44.249682 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d596299c82b4ff1ac151d68b97ff55b with delete_type TABLET_DATA_DELETED (replaced by tablet 4cdec13ee70d402fa3a08119e7bf1313) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:44.257014 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc0e5cb08c44f7b9662d63905d99b02. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:44.248428 13665 ts_tablet_manager.cc:1460] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3: Tablet startup took 732ms
W20260810 23:32:44.265054 13471 log.cc:924] Time spent T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Append to log took a long time: real 0.277s user 0.002s sys 0.000s
W20260810 23:32:44.265048 13665 ts_tablet_manager.cc:1462] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3: Trace:
0810 23:32:32.414529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:32.414812 (+ 283us) service_pool.cc:227] Handling call
0810 23:32:32.417912 (+ 3100us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:32.418112 (+ 200us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:32.734510 (+316398us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:33.307404 (+572894us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:33.536511 (+229107us) inbound_call.cc:173] Queueing success response
0810 23:32:42.899568 (+9363057us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0810 23:32:43.516368 (+616800us) ts_tablet_manager.cc:1429] Starting tablet replica
0810 23:32:43.812789 (+296421us) tablet_replica.cc:259] Starting instrumentation
0810 23:32:43.813473 (+ 684us) tablet_replica.cc:278] Starting consensus
Metrics: {"mutex_wait_us":11842,"raft.queue_time_us":131849,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":348,"tablet-open.queue_time_us":9363212,"thread_start_us":385333,"threads_started":1}
I20260810 23:32:44.271346 13058 raft_consensus.cc:2454] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
W20260810 23:32:44.113965 13691 ts_tablet_manager.cc:1462] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b: Trace:
0810 23:32:34.205189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:34.205416 (+ 227us) service_pool.cc:227] Handling call
0810 23:32:34.207929 (+ 2513us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:34.208497 (+ 568us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:34.409528 (+201031us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:34.476763 (+ 67235us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:34.478978 (+ 2215us) inbound_call.cc:173] Queueing success response
0810 23:32:43.077038 (+8598060us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0810 23:32:43.605648 (+528610us) ts_tablet_manager.cc:1429] Starting tablet replica
0810 23:32:43.829988 (+224340us) tablet_replica.cc:259] Starting instrumentation
0810 23:32:43.861392 (+ 31404us) tablet_replica.cc:278] Starting consensus
Metrics: {"mutex_wait_us":193034,"raft.queue_time_us":74235,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":313,"tablet-open.queue_time_us":8580544}
I20260810 23:32:44.285547 13691 tablet_bootstrap.cc:492] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:44.294641 13974 ts_tablet_manager.cc:1918] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:44.295614 13974 log.cc:1196] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/c45f9dfd03764d7fb1b6fb5e28b66bd3
I20260810 23:32:44.296680 13974 ts_tablet_manager.cc:1939] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
W20260810 23:32:44.307456 12958 consensus_peers.cc:487] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c45f9dfd03764d7fb1b6fb5e28b66bd3. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:44.308586 12958 consensus_peers.cc:487] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0ba75f2c2b54ec1968db8feeacf8383. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:44.317639 14346 tablet.cc:1717] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b: Skipping requested alter to schema version 2, tablet already version 2
I20260810 23:32:44.329183 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 114) took 5431 ms. Trace:
I20260810 23:32:44.329610 13974 rpcz_store.cc:276] 0810 23:32:38.898234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.898400 (+ 166us) service_pool.cc:227] Handling call
0810 23:32:43.834527 (+4936127us) ts_tablet_manager.cc:1145] Deleting tablet c45f9dfd03764d7fb1b6fb5e28b66bd3
0810 23:32:43.834602 (+ 75us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.131711 (+297109us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.294258 (+162547us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.329084 (+ 34826us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":142083,"tablet-delete.queue_time_us":4908725}
I20260810 23:32:44.331233 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 119) took 4989 ms. Trace:
I20260810 23:32:44.331555 13974 rpcz_store.cc:276] 0810 23:32:39.341702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:39.341879 (+ 177us) service_pool.cc:227] Handling call
0810 23:32:44.330709 (+4988830us) ts_tablet_manager.cc:1145] Deleting tablet 13ad5d63a2104995990a86a000963cdc
0810 23:32:44.330781 (+ 72us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.331162 (+ 381us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4983331}
I20260810 23:32:44.334396 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 123) took 4212 ms. Trace:
I20260810 23:32:44.336784 13665 tablet_bootstrap.cc:492] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:44.336623 13974 rpcz_store.cc:276] 0810 23:32:40.122283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.122816 (+ 533us) service_pool.cc:227] Handling call
0810 23:32:44.333535 (+4210719us) ts_tablet_manager.cc:1145] Deleting tablet c3d4c879ca6a402dbafe91940e7b367a
0810 23:32:44.333674 (+ 139us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.334299 (+ 625us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4072879}
I20260810 23:32:44.340158 14121 ts_tablet_manager.cc:1918] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:44.340950 14121 log.cc:1196] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/0c0d6cd2ad1d4a65929b0d856a171592
I20260810 23:32:44.342141 14121 ts_tablet_manager.cc:1939] T 0c0d6cd2ad1d4a65929b0d856a171592 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:44.344975 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 125) took 4046 ms. Trace:
I20260810 23:32:44.345379 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 103) took 5443 ms. Trace:
I20260810 23:32:44.345335 13974 rpcz_store.cc:276] 0810 23:32:40.298050 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.298283 (+ 233us) service_pool.cc:227] Handling call
0810 23:32:44.337969 (+4039686us) ts_tablet_manager.cc:1145] Deleting tablet 9c29b11192604ac18c21b7bdc0238fb3
0810 23:32:44.341651 (+ 3682us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.344271 (+ 2620us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4038938}
I20260810 23:32:44.345738 14121 rpcz_store.cc:276] 0810 23:32:38.901791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.906350 (+ 4559us) service_pool.cc:227] Handling call
0810 23:32:43.513568 (+4607218us) ts_tablet_manager.cc:1145] Deleting tablet 0c0d6cd2ad1d4a65929b0d856a171592
0810 23:32:43.513626 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.306510 (+792884us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.339671 (+ 33161us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.345290 (+ 5619us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":203646,"tablet-delete.queue_time_us":4606336}
I20260810 23:32:44.348109 14121 tablet_replica.cc:330] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:44.348495 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 127) took 3702 ms. Trace:
I20260810 23:32:44.348817 13974 rpcz_store.cc:276] 0810 23:32:40.646363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.647171 (+ 808us) service_pool.cc:227] Handling call
0810 23:32:44.346159 (+3698988us) ts_tablet_manager.cc:1145] Deleting tablet 0c0d6cd2ad1d4a65929b0d856a171592
0810 23:32:44.346803 (+ 644us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.348422 (+ 1619us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3697855}
I20260810 23:32:44.349265 14121 raft_consensus.cc:2229] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:44.349838 13974 tablet_replica.cc:330] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:44.350343 14121 raft_consensus.cc:2258] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:44.352670 14121 ts_tablet_manager.cc:1905] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:44.379988 14121 ts_tablet_manager.cc:1918] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:44.385146 14121 log.cc:1196] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/c45f9dfd03764d7fb1b6fb5e28b66bd3
I20260810 23:32:44.386400 14121 ts_tablet_manager.cc:1939] T c45f9dfd03764d7fb1b6fb5e28b66bd3 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
W20260810 23:32:44.227859 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 148137us
I20260810 23:32:44.393383 13974 raft_consensus.cc:2229] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:44.394456 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 106) took 5491 ms. Trace:
I20260810 23:32:44.395991 12700 leader_election.cc:302] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:44.397126 14278 raft_consensus.cc:2781] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:44.397861 14278 raft_consensus.cc:686] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Becoming Leader. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Running, Role: LEADER
I20260810 23:32:44.394872 14121 rpcz_store.cc:276] 0810 23:32:38.902960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.908203 (+ 5243us) service_pool.cc:227] Handling call
0810 23:32:44.347778 (+5439575us) ts_tablet_manager.cc:1145] Deleting tablet c45f9dfd03764d7fb1b6fb5e28b66bd3
0810 23:32:44.347841 (+ 63us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.363598 (+ 15757us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.379547 (+ 15949us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.394361 (+ 14814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5438767}
I20260810 23:32:44.399222 14278 consensus_queue.cc:237] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:44.409034 14121 tablet_replica.cc:330] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:44.411149 14121 raft_consensus.cc:2229] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:44.412070 14121 raft_consensus.cc:2258] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:44.421828 14121 ts_tablet_manager.cc:1905] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:44.425213 12769 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 160) took 1042 ms. Trace:
I20260810 23:32:44.364108 14352 tablet.cc:1722] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:44.432281 12698 leader_election.cc:302] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:32:44.434315 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 209888us
I20260810 23:32:44.435421 14178 raft_consensus.cc:2726] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:44.452311 13974 raft_consensus.cc:2258] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:44.460340 14343 raft_consensus.cc:504] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:44.481387 13014 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 143) took 1211 ms. Trace:
I20260810 23:32:44.471282 12769 rpcz_store.cc:276] 0810 23:32:43.382385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.383679 (+ 1294us) service_pool.cc:227] Handling call
0810 23:32:43.714552 (+330873us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:43.714644 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:44.147406 (+432762us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.424368 (+276962us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:44.425117 (+ 749us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 23:32:44.501600 14368 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.399s user 0.006s sys 0.013s
W20260810 23:32:44.496762 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 50687.8us
I20260810 23:32:44.522073 13665 tablet_bootstrap.cc:654] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:44.533119 13213 consensus_peers.cc:487] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:44.534696 13213 consensus_peers.cc:487] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd110664bac34f52bfe922a3a76c6290. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:44.536444 12763 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 155) took 1319 ms. Trace:
I20260810 23:32:44.542644 12763 rpcz_store.cc:276] 0810 23:32:43.216672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.216870 (+ 198us) service_pool.cc:227] Handling call
0810 23:32:43.713600 (+496730us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:43.713716 (+ 116us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:44.150010 (+436294us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.531715 (+381705us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:44.536364 (+ 4649us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:44.556249 13935 ts_tablet_manager.cc:1918] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:44.497327 14376 tablet.cc:1722] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:44.556855 13014 rpcz_store.cc:276] 0810 23:32:43.270132 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.271902 (+ 1770us) service_pool.cc:227] Handling call
0810 23:32:43.316986 (+ 45084us) spinlock_profiling.cc:243] Waited 872 us on lock 0x7b5800001898. stack: 00007fa86ed56033 00007fa86ed55f30 00007fa86e8d7534 00000000004db2b4 00000000004db249 00007fa86ebccf11 00007fa86ed50851 00007fa86ed50702 00007fa877be7b09 00007fa876c44730 00007fa876d26d21 00007fa876d041e3 00007fa876cb5869 00007fa8753a7c44 00007fa8753a7bd2 00007fa8753a7b01
0810 23:32:43.317179 (+ 193us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:43.317364 (+ 185us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:44.015704 (+698340us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.480531 (+464827us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:44.481300 (+ 769us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2005888}
I20260810 23:32:44.558444 13935 log.cc:1196] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/b8986e59b2314bd69b9b1de36f1f2ba7
I20260810 23:32:44.559374 13935 ts_tablet_manager.cc:1939] T b8986e59b2314bd69b9b1de36f1f2ba7 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:44.501556 13691 tablet_bootstrap.cc:654] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:44.566021 14368 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:53280 (request call id 147) took 1684 ms. Trace:
I20260810 23:32:44.508908 13020 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 148) took 1125 ms. Trace:
I20260810 23:32:44.569701 13020 rpcz_store.cc:276] 0810 23:32:43.383789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.387683 (+ 3894us) service_pool.cc:227] Handling call
0810 23:32:43.477441 (+ 89758us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:43.477600 (+ 159us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:44.030338 (+552738us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.507916 (+477578us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:44.508803 (+ 887us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:44.509527 13017 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 150) took 1044 ms. Trace:
I20260810 23:32:44.571293 13017 rpcz_store.cc:276] 0810 23:32:43.464920 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.465122 (+ 202us) service_pool.cc:227] Handling call
0810 23:32:43.468256 (+ 3134us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:43.468495 (+ 239us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:44.244919 (+776424us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.508231 (+263312us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:44.509456 (+ 1225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:44.560333 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 130) took 3766 ms. Trace:
I20260810 23:32:44.519132 13007 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 151) took 1040 ms. Trace:
I20260810 23:32:44.572407 13935 rpcz_store.cc:276] 0810 23:32:40.794208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.828401 (+ 34193us) service_pool.cc:227] Handling call
0810 23:32:44.088934 (+3260533us) ts_tablet_manager.cc:1145] Deleting tablet b8986e59b2314bd69b9b1de36f1f2ba7
0810 23:32:44.088994 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.499340 (+410346us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.555924 (+ 56584us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.560269 (+ 4345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3259755}
I20260810 23:32:44.573673 13935 tablet_replica.cc:330] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:44.572898 13007 rpcz_store.cc:276] 0810 23:32:43.479136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.483107 (+ 3971us) service_pool.cc:227] Handling call
0810 23:32:43.485593 (+ 2486us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:43.485784 (+ 191us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:44.026408 (+540624us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.518516 (+492108us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:44.519045 (+ 529us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:44.576331 13935 raft_consensus.cc:2229] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:44.577262 13974 ts_tablet_manager.cc:1905] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:44.579895 13935 raft_consensus.cc:2258] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:44.582862 13935 ts_tablet_manager.cc:1905] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:44.554219 14121 ts_tablet_manager.cc:1918] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:44.584708 14121 log.cc:1196] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/585baf44aca142199fbc268ff3bcd8f4
I20260810 23:32:44.587144 14121 ts_tablet_manager.cc:1939] T 585baf44aca142199fbc268ff3bcd8f4 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:44.566546 14368 rpcz_store.cc:276] 0810 23:32:42.881647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.884196 (+ 2549us) service_pool.cc:227] Handling call
0810 23:32:44.565900 (+1681704us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:42.887701 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 23:32:42.888822 (+ 1121us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 23:32:42.888922 (+ 100us) alter_schema_op.cc:65] Acquired schema lock
0810 23:32:42.889013 (+ 91us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 23:32:42.889262 (+ 249us) alter_schema_op.cc:126] START. Timestamp: 1786404762888798
0810 23:32:42.889355 (+ 93us) op_driver.cc:331] REPLICATION: starting
0810 23:32:42.890525 (+ 1170us) log.cc:842] Serialized 173 byte log entry
0810 23:32:43.969544 (+1079019us) op_driver.cc:444] REPLICATION: finished
0810 23:32:44.098907 (+129363us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 23:32:44.102985 (+ 4078us) log.cc:842] Serialized 5 byte log entry
0810 23:32:44.552893 (+449908us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.564636 (+ 11743us) log.cc:842] Serialized 14 byte log entry
0810 23:32:44.565733 (+ 1097us) alter_schema_op.cc:71] Released schema lock
0810 23:32:44.565769 (+ 36us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":128935,"mutex_wait_us":5821,"prepare.queue_time_us":878,"prepare.run_cpu_time_us":17774,"prepare.run_wall_time_us":160214,"raft.queue_time_us":8639,"raft.run_cpu_time_us":1204,"raft.run_wall_time_us":1203,"replication_time_us":1080085,"thread_start_us":558171,"threads_started":3,"wal-append.queue_time_us":540891,"wal-append.run_cpu_time_us":4123,"wal-append.run_wall_time_us":1037022}]]}
I20260810 23:32:44.555287 12764 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 159) took 1092 ms. Trace:
I20260810 23:32:44.627976 12764 rpcz_store.cc:276] 0810 23:32:43.381819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.382019 (+ 200us) service_pool.cc:227] Handling call
0810 23:32:43.714257 (+332238us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:43.714368 (+ 111us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:44.162505 (+448137us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.459227 (+296722us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:44.460016 (+ 789us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 23:32:44.553416 12698 leader_election.cc:341] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:44.638072 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 84739.4us
I20260810 23:32:44.638017 14346 tablet.cc:1722] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:32:44.642800 13691 tablet_bootstrap.cc:492] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:44.643468 13691 ts_tablet_manager.cc:1397] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.358s user 0.014s sys 0.010s
I20260810 23:32:44.646175 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet b8986e59b2314bd69b9b1de36f1f2ba7 (table default.Table_26bf2c7d86704b5b981ecebb4e90ba75 [id=cefe7b97c1db42ef8712bf3c7a359af2]) successfully deleted
I20260810 23:32:44.650740 14178 raft_consensus.cc:482] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:44.651327 14178 raft_consensus.cc:504] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:44.652284 14395 tablet.cc:1722] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:32:44.647989 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 0c0d6cd2ad1d4a65929b0d856a171592 (table default.Table_9e4b92615b33447baec585c92237ff05 [id=287124fe670e4041a492d9a4d4524980]) successfully deleted
I20260810 23:32:44.658270 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 102) took 5756 ms. Trace:
I20260810 23:32:44.627133 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]) successfully deleted
I20260810 23:32:44.658689 14121 rpcz_store.cc:276] 0810 23:32:38.901411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:38.907361 (+ 5950us) service_pool.cc:227] Handling call
0810 23:32:44.408627 (+5501266us) ts_tablet_manager.cc:1145] Deleting tablet 585baf44aca142199fbc268ff3bcd8f4
0810 23:32:44.408752 (+ 125us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.492213 (+ 83461us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.553907 (+ 61694us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.658176 (+104269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5493978}
I20260810 23:32:44.658345 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]) successfully deleted
I20260810 23:32:44.661680 14178 leader_election.cc:288] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:44.663702 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081b4e1f53c74f768a4bf401ae22b585" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:44.665284 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 585baf44aca142199fbc268ff3bcd8f4 (table default.Table_edb5813389c6488ea72bae6a8373f405 [id=5c3bdcf03c3d43cbbbd2b53dcdc8fa95]) successfully deleted
W20260810 23:32:44.668218 12700 leader_election.cc:341] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:44.669622 13691 raft_consensus.cc:348] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:44.670578 13691 raft_consensus.cc:374] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:44.670902 13691 raft_consensus.cc:729] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
W20260810 23:32:44.673446 12761 consensus_meta.cc:287] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.562s user 0.012s sys 0.003s
I20260810 23:32:44.673053 13691 consensus_queue.cc:260] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:44.676406 12761 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 162) took 1199 ms. Trace:
W20260810 23:32:44.660897 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 13ad5d63a2104995990a86a000963cdc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13ad5d63a2104995990a86a000963cdc already in progress: creating tablet
I20260810 23:32:44.679988 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124259164adf404c9cbe5f4e0d5e16c0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
W20260810 23:32:44.680481 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet c3d4c879ca6a402dbafe91940e7b367a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3d4c879ca6a402dbafe91940e7b367a already in progress: creating tablet
I20260810 23:32:44.680976 13062 raft_consensus.cc:3037] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:44.681578 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 9c29b11192604ac18c21b7bdc0238fb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c29b11192604ac18c21b7bdc0238fb3 already in progress: creating tablet
W20260810 23:32:44.682657 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 0c0d6cd2ad1d4a65929b0d856a171592 because the tablet was not found. No further retry: Not found: Tablet not found: 0c0d6cd2ad1d4a65929b0d856a171592
I20260810 23:32:44.672060 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 110) took 5296 ms. Trace:
I20260810 23:32:44.684257 14121 rpcz_store.cc:276] 0810 23:32:39.375430 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:39.375575 (+ 145us) service_pool.cc:227] Handling call
0810 23:32:44.667761 (+5292186us) ts_tablet_manager.cc:1145] Deleting tablet 13ad5d63a2104995990a86a000963cdc
0810 23:32:44.667827 (+ 66us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.671959 (+ 4132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5291186}
I20260810 23:32:44.686685 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 114) took 5227 ms. Trace:
I20260810 23:32:44.687184 14121 rpcz_store.cc:276] 0810 23:32:39.459616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:39.459965 (+ 349us) service_pool.cc:227] Handling call
0810 23:32:44.686182 (+5226217us) ts_tablet_manager.cc:1145] Deleting tablet c3d4c879ca6a402dbafe91940e7b367a
0810 23:32:44.686243 (+ 61us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.686615 (+ 372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5225044}
I20260810 23:32:44.694859 12223 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 226) took 1202 ms. Trace:
I20260810 23:32:44.695554 12223 rpcz_store.cc:276] 0810 23:32:43.492756 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.521916 (+ 29160us) service_pool.cc:227] Handling call
0810 23:32:43.563116 (+ 41200us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:43.565514 (+ 2398us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:44.692754 (+1127240us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:44.693646 (+ 892us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:44.694552 (+ 906us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:43.696409 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:43.696920 (+ 511us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:43.696949 (+ 29us) write_op.cc:435] Acquired schema lock
0810 23:32:43.696964 (+ 15us) tablet.cc:660] Decoding operations
0810 23:32:43.697280 (+ 316us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:43.697328 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:43.697375 (+ 47us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:43.697659 (+ 284us) tablet.cc:699] Row locks acquired
0810 23:32:43.697684 (+ 25us) write_op.cc:260] PREPARE: finished
0810 23:32:43.697854 (+ 170us) write_op.cc:270] Start()
0810 23:32:43.697960 (+ 106us) write_op.cc:276] Timestamp: P: 1786404763697799 usec, L: 0
0810 23:32:43.697982 (+ 22us) op_driver.cc:331] REPLICATION: starting
0810 23:32:43.698727 (+ 745us) log.cc:842] Serialized 433 byte log entry
0810 23:32:44.657199 (+958472us) op_driver.cc:444] REPLICATION: finished
0810 23:32:44.688397 (+ 31198us) write_op.cc:301] APPLY: starting
0810 23:32:44.688592 (+ 195us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:44.688813 (+ 221us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:44.689035 (+ 222us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:44.689847 (+ 812us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:44.690172 (+ 325us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:44.690257 (+ 85us) write_op.cc:312] APPLY: finished
0810 23:32:44.690772 (+ 515us) log.cc:842] Serialized 29 byte log entry
0810 23:32:44.691087 (+ 315us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:44.691270 (+ 183us) write_op.cc:454] Released schema lock
0810 23:32:44.691336 (+ 66us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:44.696721 (+ 5385us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:44.696762 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":30822,"apply.run_cpu_time_us":3835,"apply.run_wall_time_us":8682,"num_ops":2,"prepare.queue_time_us":129444,"prepare.run_cpu_time_us":2724,"prepare.run_wall_time_us":31887,"raft.queue_time_us":10631,"raft.run_cpu_time_us":1435,"raft.run_wall_time_us":5793,"replication_time_us":958894}]]}
I20260810 23:32:44.704145 14236 raft_consensus.cc:482] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:44.704809 14236 raft_consensus.cc:504] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:44.709102 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b261673faa964b65af3576c89e302864" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:44.709601 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 116) took 4419 ms. Trace:
I20260810 23:32:44.710048 14121 rpcz_store.cc:276] 0810 23:32:40.290795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.294388 (+ 3593us) service_pool.cc:227] Handling call
0810 23:32:44.709046 (+4414658us) ts_tablet_manager.cc:1145] Deleting tablet 9c29b11192604ac18c21b7bdc0238fb3
0810 23:32:44.709117 (+ 71us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.709515 (+ 398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4413424}
I20260810 23:32:44.711900 14236 leader_election.cc:288] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
W20260810 23:32:44.711056 12952 leader_election.cc:341] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:44.712821 14121 tablet_replica.cc:330] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:44.720780 13691 ts_tablet_manager.cc:1428] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.077s user 0.006s sys 0.001s
I20260810 23:32:44.727909 14121 raft_consensus.cc:2229] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:44.728844 14121 raft_consensus.cc:2258] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b [term 2 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:44.726317 12215 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 61157.4us
I20260810 23:32:44.737438 14121 ts_tablet_manager.cc:1905] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:44.639425 12698 leader_election.cc:341] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:44.690106 14343 leader_election.cc:288] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:44.744166 13691 tablet_bootstrap.cc:492] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:44.760500 14395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:53280 (request call id 151) took 1876 ms. Trace:
I20260810 23:32:44.764988 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:44.766748 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 88c53785fdd24df79f138df6176e75d2 (table default.Table_d0b8422dca5e40ff919741fcee7d547c [id=71c9424f327040a68889fc257935c535]). Expected version 3 got 2
I20260810 23:32:44.765998 14395 rpcz_store.cc:276] 0810 23:32:42.883927 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.912319 (+ 28392us) service_pool.cc:227] Handling call
0810 23:32:44.760401 (+1848082us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:43.048210 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 23:32:43.048961 (+ 751us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 23:32:44.580136 (+1531175us) alter_schema_op.cc:65] Acquired schema lock
0810 23:32:44.580219 (+ 83us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 23:32:44.580403 (+ 184us) alter_schema_op.cc:126] START. Timestamp: 1786404764580292
0810 23:32:44.580453 (+ 50us) op_driver.cc:331] REPLICATION: starting
0810 23:32:44.581212 (+ 759us) log.cc:842] Serialized 173 byte log entry
0810 23:32:44.592786 (+ 11574us) op_driver.cc:444] REPLICATION: finished
0810 23:32:44.622094 (+ 29308us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 23:32:44.654032 (+ 31938us) log.cc:842] Serialized 5 byte log entry
0810 23:32:44.731791 (+ 77759us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.756802 (+ 25011us) log.cc:842] Serialized 14 byte log entry
0810 23:32:44.760182 (+ 3380us) alter_schema_op.cc:71] Released schema lock
0810 23:32:44.760218 (+ 36us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":28915,"prepare.queue_time_us":134341,"prepare.run_cpu_time_us":52755,"prepare.run_wall_time_us":1553865,"raft.queue_time_us":117,"raft.run_cpu_time_us":745,"raft.run_wall_time_us":745,"replication_time_us":12233,"thread_start_us":27387,"threads_started":1}]]}
W20260810 23:32:44.744071 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 13ad5d63a2104995990a86a000963cdc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13ad5d63a2104995990a86a000963cdc already in progress: creating tablet
W20260810 23:32:44.777161 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet c3d4c879ca6a402dbafe91940e7b367a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3d4c879ca6a402dbafe91940e7b367a already in progress: creating tablet
W20260810 23:32:44.778182 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 9c29b11192604ac18c21b7bdc0238fb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c29b11192604ac18c21b7bdc0238fb3 already in progress: creating tablet
W20260810 23:32:44.748935 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 109541us
W20260810 23:32:44.779664 12698 leader_election.cc:341] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:44.780727 13691 tablet_bootstrap.cc:654] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:44.746840 12761 rpcz_store.cc:276] 0810 23:32:43.477298 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.477487 (+ 189us) service_pool.cc:227] Handling call
0810 23:32:43.714094 (+236607us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:43.714198 (+ 104us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:44.110843 (+396645us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.675585 (+564742us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:44.676310 (+ 725us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:44.695358 14391 tablet.cc:1722] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:44.546491 13514 tablet_bootstrap.cc:654] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:44.801162 14343 raft_consensus.cc:482] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:44.801908 14343 raft_consensus.cc:504] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:44.805653 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b261673faa964b65af3576c89e302864" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:44.806540 13058 raft_consensus.cc:2454] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:32:44.808373 12700 leader_election.cc:302] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:44.809722 14278 raft_consensus.cc:2781] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:44.810230 14278 raft_consensus.cc:482] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:44.810665 14278 raft_consensus.cc:3037] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:44.780740 12698 leader_election.cc:302] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:44.813273 14178 raft_consensus.cc:2726] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:44.825799 14343 leader_election.cc:288] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:44.830911 13665 tablet_bootstrap.cc:492] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:44.828200 13935 ts_tablet_manager.cc:1918] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:44.840868 13935 log.cc:1196] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/18b29cda94584badb1225477ec2bd1bc
I20260810 23:32:44.846278 13691 tablet_bootstrap.cc:492] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:44.842729 13935 ts_tablet_manager.cc:1939] T 18b29cda94584badb1225477ec2bd1bc P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:44.850131 13691 ts_tablet_manager.cc:1397] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.106s user 0.014s sys 0.000s
W20260810 23:32:44.851168 12237 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fb2ee1c9b41478da001aa67d7a7ad3c consensus state has no leader
I20260810 23:32:44.852010 12237 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 227) took 1286 ms. Trace:
W20260810 23:32:44.857101 12228 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e62dc27b59ba469d99e8d11372aceff5 has no consensus state
I20260810 23:32:44.857355 14278 raft_consensus.cc:504] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:44.858177 12228 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 228) took 1292 ms. Trace:
I20260810 23:32:44.856877 13062 raft_consensus.cc:2454] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:44.858631 12228 rpcz_store.cc:276] 0810 23:32:43.565917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.566093 (+ 176us) service_pool.cc:227] Handling call
0810 23:32:43.569328 (+ 3235us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:43.570945 (+ 1617us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:43.578663 (+ 7718us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:43.578930 (+ 267us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:44.786982 (+1208052us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:44.858078 (+ 71096us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:43.769482 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:43.769884 (+ 402us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:43.769909 (+ 25us) write_op.cc:435] Acquired schema lock
0810 23:32:43.769922 (+ 13us) tablet.cc:660] Decoding operations
0810 23:32:43.770227 (+ 305us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:43.770269 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:43.770296 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:43.770598 (+ 302us) tablet.cc:699] Row locks acquired
0810 23:32:43.770618 (+ 20us) write_op.cc:260] PREPARE: finished
0810 23:32:43.770757 (+ 139us) write_op.cc:270] Start()
0810 23:32:43.770868 (+ 111us) write_op.cc:276] Timestamp: P: 1786404763770701 usec, L: 0
0810 23:32:43.770886 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 23:32:43.771487 (+ 601us) log.cc:842] Serialized 405 byte log entry
0810 23:32:44.781594 (+1010107us) op_driver.cc:444] REPLICATION: finished
0810 23:32:44.784214 (+ 2620us) write_op.cc:301] APPLY: starting
0810 23:32:44.784300 (+ 86us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:44.784438 (+ 138us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:44.784559 (+ 121us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:44.784770 (+ 211us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:44.784924 (+ 154us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:44.784970 (+ 46us) write_op.cc:312] APPLY: finished
0810 23:32:44.785174 (+ 204us) log.cc:842] Serialized 23 byte log entry
0810 23:32:44.785362 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:44.785435 (+ 73us) write_op.cc:454] Released schema lock
0810 23:32:44.785467 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:44.787340 (+ 1873us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:44.787408 (+ 68us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2435,"apply.run_cpu_time_us":1972,"apply.run_wall_time_us":3551,"num_ops":1,"prepare.queue_time_us":188108,"prepare.run_cpu_time_us":2376,"prepare.run_wall_time_us":33477,"raft.queue_time_us":102,"raft.run_cpu_time_us":771,"raft.run_wall_time_us":781,"replication_time_us":1010582,"thread_start_us":32306,"threads_started":1}]]}
I20260810 23:32:44.865808 13691 raft_consensus.cc:348] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:44.880728 13691 raft_consensus.cc:374] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:44.887882 13691 raft_consensus.cc:729] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:44.893873 13691 consensus_queue.cc:260] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:44.899956 14121 ts_tablet_manager.cc:1918] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:44.900527 14121 log.cc:1196] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/1f791b2df0524997ab1c53a9e5cb39ca
I20260810 23:32:44.901325 14121 ts_tablet_manager.cc:1939] T 1f791b2df0524997ab1c53a9e5cb39ca P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:44.902302 13665 ts_tablet_manager.cc:1397] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.566s user 0.018s sys 0.016s
I20260810 23:32:44.905313 13665 raft_consensus.cc:348] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:44.906088 13665 raft_consensus.cc:374] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:44.906345 13665 raft_consensus.cc:729] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:44.907014 13665 consensus_queue.cc:260] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:44.921470 13691 ts_tablet_manager.cc:1428] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.066s user 0.007s sys 0.000s
I20260810 23:32:44.889118 12235 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 230) took 1132 ms. Trace:
I20260810 23:32:44.919673 12224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 806fe5162aef4d36ad5e2ac324047426
I20260810 23:32:44.926318 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 118) took 4392 ms. Trace:
I20260810 23:32:44.927901 13273 tablet_service.cc:1514] Processing DeleteTablet for tablet 806fe5162aef4d36ad5e2ac324047426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:44.924010 12235 rpcz_store.cc:276] 0810 23:32:43.756784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.756921 (+ 137us) service_pool.cc:227] Handling call
0810 23:32:43.760256 (+ 3335us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:43.762082 (+ 1826us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:44.887714 (+1125632us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:44.888324 (+ 610us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:44.888887 (+ 563us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:43.842334 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:43.842994 (+ 660us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:43.843023 (+ 29us) write_op.cc:435] Acquired schema lock
0810 23:32:43.843036 (+ 13us) tablet.cc:660] Decoding operations
0810 23:32:43.843361 (+ 325us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:43.843407 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:43.843430 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:43.843664 (+ 234us) tablet.cc:699] Row locks acquired
0810 23:32:43.843679 (+ 15us) write_op.cc:260] PREPARE: finished
0810 23:32:43.843832 (+ 153us) write_op.cc:270] Start()
0810 23:32:43.843977 (+ 145us) write_op.cc:276] Timestamp: P: 1786404763843767 usec, L: 0
0810 23:32:43.843996 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 23:32:43.865112 (+ 21116us) log.cc:842] Serialized 433 byte log entry
0810 23:32:44.885308 (+1020196us) op_driver.cc:444] REPLICATION: finished
0810 23:32:44.885667 (+ 359us) write_op.cc:301] APPLY: starting
0810 23:32:44.885744 (+ 77us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:44.885901 (+ 157us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:44.886021 (+ 120us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:44.886523 (+ 502us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:44.886687 (+ 164us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:44.886736 (+ 49us) write_op.cc:312] APPLY: finished
0810 23:32:44.886986 (+ 250us) log.cc:842] Serialized 29 byte log entry
0810 23:32:44.887155 (+ 169us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:44.887239 (+ 84us) write_op.cc:454] Released schema lock
0810 23:32:44.887274 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:44.889572 (+ 2298us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:44.889605 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":92,"apply.run_cpu_time_us":2089,"apply.run_wall_time_us":4119,"num_ops":2,"prepare.queue_time_us":78965,"prepare.run_cpu_time_us":3534,"prepare.run_wall_time_us":42362,"raft.queue_time_us":8819,"raft.run_cpu_time_us":1108,"raft.run_wall_time_us":1109,"replication_time_us":1041153}]]}
I20260810 23:32:44.926865 14121 rpcz_store.cc:276] 0810 23:32:40.533704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.533919 (+ 215us) service_pool.cc:227] Handling call
0810 23:32:44.712618 (+4178699us) ts_tablet_manager.cc:1145] Deleting tablet 1f791b2df0524997ab1c53a9e5cb39ca
0810 23:32:44.712673 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.836972 (+124299us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.883520 (+ 46548us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.926232 (+ 42712us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13891,"tablet-delete.queue_time_us":4177926}
I20260810 23:32:44.934015 14121 tablet_replica.cc:330] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:44.926733 12224 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 231) took 1092 ms. Trace:
I20260810 23:32:44.934922 14121 raft_consensus.cc:2229] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:44.935456 14121 raft_consensus.cc:2258] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:44.934808 12224 rpcz_store.cc:276] 0810 23:32:43.833958 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.834331 (+ 373us) service_pool.cc:227] Handling call
0810 23:32:43.836421 (+ 2090us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:43.837146 (+ 725us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:43.837334 (+ 188us) catalog_manager.cc:2761] Locking tablets
0810 23:32:43.838052 (+ 718us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:44.918876 (+1080824us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:44.919257 (+ 381us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:44.919408 (+ 151us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:44.926603 (+ 7195us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:43.884871 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:43.885407 (+ 536us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:43.885440 (+ 33us) write_op.cc:435] Acquired schema lock
0810 23:32:43.885459 (+ 19us) tablet.cc:660] Decoding operations
0810 23:32:43.885894 (+ 435us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:43.885944 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:43.885988 (+ 44us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:43.894336 (+ 8348us) tablet.cc:699] Row locks acquired
0810 23:32:43.894357 (+ 21us) write_op.cc:260] PREPARE: finished
0810 23:32:43.894556 (+ 199us) write_op.cc:270] Start()
0810 23:32:43.894672 (+ 116us) write_op.cc:276] Timestamp: P: 1786404763894455 usec, L: 0
0810 23:32:43.894692 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:32:43.895414 (+ 722us) log.cc:842] Serialized 734 byte log entry
0810 23:32:44.898391 (+1002977us) op_driver.cc:444] REPLICATION: finished
0810 23:32:44.916938 (+ 18547us) write_op.cc:301] APPLY: starting
0810 23:32:44.917070 (+ 132us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:44.917236 (+ 166us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:44.917400 (+ 164us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:44.917771 (+ 371us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:44.917933 (+ 162us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:44.917988 (+ 55us) write_op.cc:312] APPLY: finished
0810 23:32:44.918245 (+ 257us) log.cc:842] Serialized 29 byte log entry
0810 23:32:44.918429 (+ 184us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:44.918521 (+ 92us) write_op.cc:454] Released schema lock
0810 23:32:44.918560 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:44.921037 (+ 2477us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:44.921070 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":18206,"apply.run_cpu_time_us":2245,"apply.run_wall_time_us":4464,"num_ops":2,"prepare.queue_time_us":43559,"prepare.run_cpu_time_us":3164,"prepare.run_wall_time_us":29191,"raft.queue_time_us":7130,"raft.run_cpu_time_us":1000,"raft.run_wall_time_us":1001,"replication_time_us":1003525}]]}
I20260810 23:32:44.941633 12224 master_service.cc:946] ReplaceTablet: received request to replace tablet ca0dd19112504f8ca57f64dcf0bd50a8 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:44.943323 14121 ts_tablet_manager.cc:1905] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:44.945070 13691 tablet_bootstrap.cc:492] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:44.957259 12228 master_service.cc:946] ReplaceTablet: received request to replace tablet 4cdec13ee70d402fa3a08119e7bf1313 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:44.852566 12237 rpcz_store.cc:276] 0810 23:32:43.565597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.567592 (+ 1995us) service_pool.cc:227] Handling call
0810 23:32:43.572670 (+ 5078us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:43.573478 (+ 808us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:43.574989 (+ 1511us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:43.575248 (+ 259us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:44.786362 (+1211114us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:44.851868 (+ 65506us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:43.728447 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:43.728893 (+ 446us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:43.728925 (+ 32us) write_op.cc:435] Acquired schema lock
0810 23:32:43.728941 (+ 16us) tablet.cc:660] Decoding operations
0810 23:32:43.729267 (+ 326us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:43.729312 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:43.729336 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:43.729508 (+ 172us) tablet.cc:699] Row locks acquired
0810 23:32:43.729521 (+ 13us) write_op.cc:260] PREPARE: finished
0810 23:32:43.729655 (+ 134us) write_op.cc:270] Start()
0810 23:32:43.729749 (+ 94us) write_op.cc:276] Timestamp: P: 1786404763729601 usec, L: 0
0810 23:32:43.729765 (+ 16us) op_driver.cc:331] REPLICATION: starting
0810 23:32:43.730459 (+ 694us) log.cc:842] Serialized 405 byte log entry
0810 23:32:44.780264 (+1049805us) op_driver.cc:444] REPLICATION: finished
0810 23:32:44.782209 (+ 1945us) write_op.cc:301] APPLY: starting
0810 23:32:44.782299 (+ 90us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:44.782445 (+ 146us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:44.782561 (+ 116us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:44.782787 (+ 226us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:44.782936 (+ 149us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:44.782986 (+ 50us) write_op.cc:312] APPLY: finished
0810 23:32:44.783214 (+ 228us) log.cc:842] Serialized 23 byte log entry
0810 23:32:44.783399 (+ 185us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:44.783498 (+ 99us) write_op.cc:454] Released schema lock
0810 23:32:44.783534 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:44.783776 (+ 242us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:44.783817 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1702,"apply.run_cpu_time_us":1827,"apply.run_wall_time_us":1828,"num_ops":1,"prepare.queue_time_us":151531,"prepare.run_cpu_time_us":2376,"prepare.run_wall_time_us":40899,"raft.queue_time_us":30139,"raft.run_cpu_time_us":871,"raft.run_wall_time_us":874,"replication_time_us":1050315}]]}
I20260810 23:32:44.961261 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_4612ad4a086f41b28be40a81f4b0a41c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:44.964442 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4612ad4a086f41b28be40a81f4b0a41c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:44.993897 13974 ts_tablet_manager.cc:1918] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:44.994659 13974 log.cc:1196] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/81e26fe4d42445279f12bb17b2841d08
I20260810 23:32:44.995877 13974 ts_tablet_manager.cc:1939] T 81e26fe4d42445279f12bb17b2841d08 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:44.997260 13691 tablet_bootstrap.cc:654] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:44.968582 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 188835us
I20260810 23:32:45.003098 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 130) took 4353 ms. Trace:
I20260810 23:32:44.969223 12700 leader_election.cc:302] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:45.010281 14178 raft_consensus.cc:2781] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:45.010010 14395 tablet.cc:1722] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:45.010928 14178 raft_consensus.cc:686] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Becoming Leader. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Running, Role: LEADER
W20260810 23:32:44.968150 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 66586.4us
I20260810 23:32:44.974735 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet 806fe5162aef4d36ad5e2ac324047426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:45.015419 14178 consensus_queue.cc:237] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.019739 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081b4e1f53c74f768a4bf401ae22b585" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:45.020535 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124259164adf404c9cbe5f4e0d5e16c0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:45.020567 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b261673faa964b65af3576c89e302864" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:45.000384 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 131) took 4170 ms. Trace:
I20260810 23:32:45.017858 14278 leader_election.cc:288] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:45.022253 13665 ts_tablet_manager.cc:1428] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.120s user 0.006s sys 0.000s
I20260810 23:32:45.022451 13935 rpcz_store.cc:276] 0810 23:32:40.829978 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.840294 (+ 10316us) service_pool.cc:227] Handling call
0810 23:32:44.573496 (+3733202us) ts_tablet_manager.cc:1145] Deleting tablet 18b29cda94584badb1225477ec2bd1bc
0810 23:32:44.573545 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.616810 (+ 43265us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.827757 (+210947us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.000013 (+172256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3731744}
I20260810 23:32:45.025316 13935 tablet_replica.cc:330] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:45.026345 13665 tablet_bootstrap.cc:492] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:44.983934 12223 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet bd77526f1db4455291fde8a001092a09 (table default.Table_ac23e0995fcd414ead1b88406cf7d79e [id=963512beef0d4063b8e6a5738abd1d92]). Expected version 2 got 0
I20260810 23:32:45.029764 12223 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:45.038565 13935 raft_consensus.cc:2229] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:45.039471 13935 raft_consensus.cc:2258] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:45.012578 13471 log.cc:924] Time spent T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Append to log took a long time: real 0.137s user 0.003s sys 0.004s
W20260810 23:32:45.015399 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (table default.Table_f577b98edfc0492cb8edbc6198d6bcd1 [id=dae244793fd24816a8199c38f23a25b5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8fb2ee1c9b41478da001aa67d7a7ad3c consensus state has no leader
I20260810 23:32:45.025326 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 18b29cda94584badb1225477ec2bd1bc (table default.Table_b6ab31c2c3cb48fe84b341db2abba9ff [id=74d3cc5da04c4e098ac075abf30a8a72]) successfully deleted
I20260810 23:32:45.043843 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 1f791b2df0524997ab1c53a9e5cb39ca (table default.Table_089a1834ee404319b9d766443f10704f [id=a74d0ca6962e44c5801d194da36bf261]) successfully deleted
I20260810 23:32:45.013103 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b261673faa964b65af3576c89e302864" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:45.048707 13062 raft_consensus.cc:3037] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:45.016917 13007 tablet_service.cc:1514] Processing DeleteTablet for tablet 806fe5162aef4d36ad5e2ac324047426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:45.034286 14121 ts_tablet_manager.cc:1918] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:45.053247 14121 log.cc:1196] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/fd110664bac34f52bfe922a3a76c6290
I20260810 23:32:45.054100 14121 ts_tablet_manager.cc:1939] T fd110664bac34f52bfe922a3a76c6290 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:45.057858 14307 raft_consensus.cc:482] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:45.058410 14307 raft_consensus.cc:504] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.060925 14307 leader_election.cc:288] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:45.062825 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840400119d20424ea0f03f2bff550d44" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:45.063556 13058 raft_consensus.cc:2454] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:45.065011 13212 leader_election.cc:302] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:45.065178 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 119) took 4525 ms. Trace:
I20260810 23:32:45.066140 14307 raft_consensus.cc:2781] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:45.066597 14307 raft_consensus.cc:482] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:45.065725 14121 rpcz_store.cc:276] 0810 23:32:40.539465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.540065 (+ 600us) service_pool.cc:227] Handling call
0810 23:32:44.933812 (+4393747us) ts_tablet_manager.cc:1145] Deleting tablet fd110664bac34f52bfe922a3a76c6290
0810 23:32:44.933865 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.988975 (+ 55110us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.034012 (+ 45037us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.065106 (+ 31094us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4392768}
I20260810 23:32:45.067015 14307 raft_consensus.cc:3037] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:45.067880 14121 tablet_replica.cc:330] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:45.068833 14121 raft_consensus.cc:2229] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:45.069422 14121 raft_consensus.cc:2258] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:45.073817 14121 ts_tablet_manager.cc:1905] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:45.077440 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet fd110664bac34f52bfe922a3a76c6290 (table default.Table_0af7ca97e51c47a29d5dd8867b152066 [id=32144cd75df84cfabce64657802c111a]) successfully deleted
I20260810 23:32:45.081518 14307 raft_consensus.cc:504] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.038618 12223 catalog_manager.cc:5495] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b reported cstate change: term changed from 0 to 1, leader changed from <none> to f26517150f1c480a877f15aec71d020b (127.11.226.130). New cstate: current_term: 1 leader_uuid: "f26517150f1c480a877f15aec71d020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:45.003443 13974 rpcz_store.cc:276] 0810 23:32:40.649792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.650358 (+ 566us) service_pool.cc:227] Handling call
0810 23:32:44.349660 (+3699302us) ts_tablet_manager.cc:1145] Deleting tablet 81e26fe4d42445279f12bb17b2841d08
0810 23:32:44.349721 (+ 61us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:44.427997 (+ 78276us) spinlock_profiling.cc:243] Waited 42.7 ms on lock 0x7b54000eae54. stack: 00007f8764d74033 00007f8764d73f30 00007f87648f5534 00000000004db2b4 00000000004db249 00007f876bc80471 00007f876bd03352 00007f876c11926c 00007f876cd26cfa 00007f876cd56e10 00007f876cd3de0d 00007f876cd3ddb9 00007f876cd3dd49 00007f876cd3dd11 00007f876cd3cf3d 0000000000580784
0810 23:32:44.720299 (+292302us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.993596 (+273297us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.003002 (+ 9406us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17960,"spinlock_wait_cycles":98217344,"tablet-delete.queue_time_us":3698644}
I20260810 23:32:45.092149 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 81e26fe4d42445279f12bb17b2841d08 (table default.Table_adf76678e17a494185985a544def0f2d [id=5a377948adc7475c8c98141deaf1c274]) successfully deleted
I20260810 23:32:45.098261 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840400119d20424ea0f03f2bff550d44" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:45.099249 13058 raft_consensus.cc:3037] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:45.041801 14352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:47546 (request call id 150) took 2083 ms. Trace:
I20260810 23:32:45.044940 13514 tablet_bootstrap.cc:492] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:45.065361 14343 raft_consensus.cc:482] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:45.102649 13974 tablet_replica.cc:330] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:45.108507 12229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 4d697e7533af4fc0a15cb41f3fcce4db
I20260810 23:32:45.108518 13514 ts_tablet_manager.cc:1397] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 1.135s user 0.012s sys 0.012s
I20260810 23:32:45.110358 12229 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 229) took 1514 ms. Trace:
I20260810 23:32:45.100359 14352 rpcz_store.cc:276] 0810 23:32:42.957942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.958247 (+ 305us) service_pool.cc:227] Handling call
0810 23:32:45.041493 (+2083246us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:43.000606 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 23:32:43.001186 (+ 580us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 23:32:43.001317 (+ 131us) alter_schema_op.cc:65] Acquired schema lock
0810 23:32:43.001397 (+ 80us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 23:32:43.001614 (+ 217us) alter_schema_op.cc:126] START. Timestamp: 1786404763001504
0810 23:32:43.001649 (+ 35us) op_driver.cc:331] REPLICATION: starting
0810 23:32:43.002547 (+ 898us) log.cc:842] Serialized 173 byte log entry
0810 23:32:43.699771 (+697224us) op_driver.cc:444] REPLICATION: finished
0810 23:32:44.187827 (+488056us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 23:32:44.445577 (+257750us) log.cc:842] Serialized 5 byte log entry
0810 23:32:44.852357 (+406780us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.853387 (+ 1030us) log.cc:842] Serialized 14 byte log entry
0810 23:32:45.041123 (+187736us) alter_schema_op.cc:71] Released schema lock
0810 23:32:45.041200 (+ 77us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":487642,"prepare.queue_time_us":40553,"prepare.run_cpu_time_us":3211,"prepare.run_wall_time_us":76497,"raft.queue_time_us":10100,"raft.run_cpu_time_us":8589,"raft.run_wall_time_us":501753,"replication_time_us":697977,"thread_start_us":565375,"threads_started":3,"wal-append.queue_time_us":2540}]]}
I20260810 23:32:45.114497 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d697e7533af4fc0a15cb41f3fcce4db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:45.114378 14343 raft_consensus.cc:504] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.117692 14343 leader_election.cc:288] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:45.119504 13514 raft_consensus.cc:348] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:45.121071 13514 raft_consensus.cc:374] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:45.121487 13514 raft_consensus.cc:729] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:45.123960 13514 consensus_queue.cc:260] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:45.127642 13514 ts_tablet_manager.cc:1428] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.019s user 0.006s sys 0.002s
I20260810 23:32:45.128998 13514 tablet_bootstrap.cc:492] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:45.136996 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b261673faa964b65af3576c89e302864" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:45.110728 12229 rpcz_store.cc:276] 0810 23:32:43.595943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.618276 (+ 22333us) service_pool.cc:227] Handling call
0810 23:32:43.623563 (+ 5287us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:43.624531 (+ 968us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:43.624740 (+ 209us) catalog_manager.cc:2761] Locking tablets
0810 23:32:43.625418 (+ 678us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:44.844352 (+1218934us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:44.850297 (+ 5945us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:45.103635 (+253338us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:45.110274 (+ 6639us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:43.803145 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:43.803569 (+ 424us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:43.803597 (+ 28us) write_op.cc:435] Acquired schema lock
0810 23:32:43.803610 (+ 13us) tablet.cc:660] Decoding operations
0810 23:32:43.804012 (+ 402us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:43.804080 (+ 68us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:43.804103 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:43.804554 (+ 451us) tablet.cc:699] Row locks acquired
0810 23:32:43.804576 (+ 22us) write_op.cc:260] PREPARE: finished
0810 23:32:43.804725 (+ 149us) write_op.cc:270] Start()
0810 23:32:43.804827 (+ 102us) write_op.cc:276] Timestamp: P: 1786404763804663 usec, L: 0
0810 23:32:43.804845 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 23:32:43.805522 (+ 677us) log.cc:842] Serialized 765 byte log entry
0810 23:32:44.841585 (+1036063us) op_driver.cc:444] REPLICATION: finished
0810 23:32:44.842048 (+ 463us) write_op.cc:301] APPLY: starting
0810 23:32:44.842147 (+ 99us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:44.842299 (+ 152us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:44.842425 (+ 126us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:44.843022 (+ 597us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:44.843207 (+ 185us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:44.843255 (+ 48us) write_op.cc:312] APPLY: finished
0810 23:32:44.843555 (+ 300us) log.cc:842] Serialized 29 byte log entry
0810 23:32:44.843713 (+ 158us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:44.843816 (+ 103us) write_op.cc:454] Released schema lock
0810 23:32:44.843852 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:44.844634 (+ 782us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:44.844662 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":226,"apply.run_cpu_time_us":2414,"apply.run_wall_time_us":3013,"num_ops":2,"prepare.queue_time_us":175172,"prepare.run_cpu_time_us":2764,"prepare.run_wall_time_us":39049,"raft.queue_time_us":35309,"raft.run_cpu_time_us":821,"raft.run_wall_time_us":821,"replication_time_us":1036555}]]}
I20260810 23:32:45.138407 12805 raft_consensus.cc:2379] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate 191d120f4c7f44fc8475305a6d2e75f3 in this term.
I20260810 23:32:45.140324 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_e0909dd9de4c461dae94163efea3a62f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:45.142603 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0909dd9de4c461dae94163efea3a62f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:45.143788 12229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_90a489ddbc1e4f7bb710756a1c40465f" } modify_external_catalogs: true
I20260810 23:32:45.144488 13935 ts_tablet_manager.cc:1905] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:45.144649 12229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_90a489ddbc1e4f7bb710756a1c40465f" } modify_external_catalogs: true
W20260810 23:32:45.148416 12234 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 29) took 1866 ms (client timeout 1000 ms). Trace:
W20260810 23:32:45.148854 12234 rpcz_store.cc:269] 0810 23:32:43.281661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.333002 (+ 51341us) service_pool.cc:227] Handling call
0810 23:32:43.448883 (+115881us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:45.148045 (+1699162us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":12,"spinlock_wait_cycles":257436928}
I20260810 23:32:45.160239 12234 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_98eeae0f1e684e5393147f22149b771a" } new_table_name: "default.Table_6c7fe24265af4acaab757c9857a87a7e" modify_external_catalogs: true
I20260810 23:32:45.162776 12240 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_d4fad7cecea149e88811df34c8c11d2d" } new_table_name: "default.Table_7f6d5cd3228d430f9e4a4376d055f8d0" modify_external_catalogs: true
I20260810 23:32:45.104380 14307 leader_election.cc:288] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:45.114902 13273 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d697e7533af4fc0a15cb41f3fcce4db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:45.121937 13278 tablet_service.cc:1514] Processing DeleteTablet for tablet 840400119d20424ea0f03f2bff550d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:45.122907 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e26fe4d42445279f12bb17b2841d08 with delete_type TABLET_DATA_DELETED (replaced by tablet b261673faa964b65af3576c89e302864) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:45.152055 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b261673faa964b65af3576c89e302864" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:45.158748 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f935119e4d9b4accb79db98185c80aca" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:45.167198 13338 raft_consensus.cc:2454] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
W20260810 23:32:45.169571 13213 leader_election.cc:341] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 840400119d20424ea0f03f2bff550d44
W20260810 23:32:45.171216 13213 leader_election.cc:341] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 840400119d20424ea0f03f2bff550d44
I20260810 23:32:45.171931 13974 raft_consensus.cc:2229] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
W20260810 23:32:45.182410 12958 consensus_peers.cc:487] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18b29cda94584badb1225477ec2bd1bc. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:45.183346 12958 leader_election.cc:302] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:45.176373 14121 ts_tablet_manager.cc:1918] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:45.184650 14121 log.cc:1196] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/81e26fe4d42445279f12bb17b2841d08
I20260810 23:32:45.186306 13974 raft_consensus.cc:2258] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:45.190014 14121 ts_tablet_manager.cc:1939] T 81e26fe4d42445279f12bb17b2841d08 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
W20260810 23:32:45.212510 12698 leader_election.cc:341] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:45.213872 12698 leader_election.cc:302] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:32:45.214744 12698 raft_consensus.cc:2670] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260810 23:32:45.215749 12698 leader_election.cc:341] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:45.217049 12698 leader_election.cc:341] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:45.216996 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f935119e4d9b4accb79db98185c80aca" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:45.217685 13054 raft_consensus.cc:2454] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:32:45.212821 13017 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d2f8688fc6046d58009800c5e288302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:38 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:45.213738 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet fd110664bac34f52bfe922a3a76c6290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:45.214290 13014 tablet_service.cc:1514] Processing DeleteTablet for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d1b9d20d09c4d1ba9a4b10959807e50) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:45.206163 14236 consensus_queue.cc:1035] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260810 23:32:45.215109 13007 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d697e7533af4fc0a15cb41f3fcce4db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:45.241230 13691 tablet_bootstrap.cc:492] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:45.243512 13691 ts_tablet_manager.cc:1397] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.303s user 0.029s sys 0.005s
I20260810 23:32:45.197266 13062 raft_consensus.cc:2454] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:45.244585 14216 raft_consensus.cc:2726] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:45.191116 13665 tablet_bootstrap.cc:700] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent opening tablet: real 0.112s user 0.003s sys 0.000s
I20260810 23:32:45.247227 13665 tablet_bootstrap.cc:654] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:45.231658 13058 raft_consensus.cc:2454] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:45.251012 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 120) took 4711 ms. Trace:
I20260810 23:32:45.251012 13212 leader_election.cc:302] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:45.251543 14121 rpcz_store.cc:276] 0810 23:32:40.539869 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.541431 (+ 1562us) service_pool.cc:227] Handling call
0810 23:32:45.067692 (+4526261us) ts_tablet_manager.cc:1145] Deleting tablet 81e26fe4d42445279f12bb17b2841d08
0810 23:32:45.067748 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:45.125806 (+ 58058us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.175474 (+ 49668us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.250829 (+ 75355us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9395,"tablet-delete.queue_time_us":4525480}
I20260810 23:32:45.254868 13974 ts_tablet_manager.cc:1905] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:45.257959 14121 tablet_replica.cc:330] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b: stopping tablet replica
W20260810 23:32:45.258217 12698 leader_election.cc:341] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:45.240548 14391 tablet.cc:1722] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:32:45.259282 14121 raft_consensus.cc:2229] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:45.229501 12700 leader_election.cc:302] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:45.260777 14178 raft_consensus.cc:2781] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:45.261224 14178 raft_consensus.cc:482] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:45.261704 14178 raft_consensus.cc:3037] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:45.263310 14121 raft_consensus.cc:2258] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:45.263664 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 81e26fe4d42445279f12bb17b2841d08 (table default.Table_adf76678e17a494185985a544def0f2d [id=5a377948adc7475c8c98141deaf1c274]) successfully deleted
I20260810 23:32:45.270004 14121 ts_tablet_manager.cc:1905] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:45.256263 13691 raft_consensus.cc:348] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.273214 13691 raft_consensus.cc:374] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:45.273620 13691 raft_consensus.cc:729] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:45.274633 13691 consensus_queue.cc:260] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.280975 13514 tablet_bootstrap.cc:654] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:45.283380 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_1072812b5a8a49a1a6a1a94eb4e2b319"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:45.290989 13691 ts_tablet_manager.cc:1428] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.047s user 0.007s sys 0.000s
W20260810 23:32:45.285283 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1072812b5a8a49a1a6a1a94eb4e2b319 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:45.296183 14307 raft_consensus.cc:2781] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:45.296840 14307 raft_consensus.cc:686] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:45.297863 14307 consensus_queue.cc:237] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:45.300112 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 45181.9us
I20260810 23:32:45.302114 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_c5dbbcbf87314046b7357e16f8b94e35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:45.304102 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5dbbcbf87314046b7357e16f8b94e35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:45.305788 12229 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 10f55e48c1a94709b583b6afd26853fa
I20260810 23:32:45.308528 13691 tablet_bootstrap.cc:492] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
W20260810 23:32:45.308729 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 50569.6us
I20260810 23:32:45.310969 12700 leader_election.cc:302] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:45.312435 14417 raft_consensus.cc:2781] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:45.313009 14417 raft_consensus.cc:686] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Becoming Leader. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Running, Role: LEADER
I20260810 23:32:45.313860 14417 consensus_queue.cc:237] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:45.323601 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 53763.8us
W20260810 23:32:45.324936 12240 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 684caf5aaa8347e8b4e02323318fb0b7 consensus state has no leader
I20260810 23:32:45.352092 13665 tablet_bootstrap.cc:492] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:45.363343 13665 ts_tablet_manager.cc:1397] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.337s user 0.014s sys 0.002s
I20260810 23:32:45.385871 13665 raft_consensus.cc:348] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.386664 13665 raft_consensus.cc:374] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:45.386947 13665 raft_consensus.cc:729] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:45.391382 12229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_addbc3e258944d0cb5d2e7fb11214412" } modify_external_catalogs: true
I20260810 23:32:45.392067 12229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_addbc3e258944d0cb5d2e7fb11214412" } modify_external_catalogs: true
I20260810 23:32:45.393695 12236 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
I20260810 23:32:45.399925 13691 tablet_bootstrap.cc:654] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:45.403963 14178 raft_consensus.cc:504] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:45.407974 12234 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ff43927ed204bfda6f647808b44fa3a has no consensus state
I20260810 23:32:45.424944 14178 leader_election.cc:288] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:45.427129 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f935119e4d9b4accb79db98185c80aca" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:45.387709 13665 consensus_queue.cc:260] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.431459 13058 raft_consensus.cc:3037] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:45.432591 13665 ts_tablet_manager.cc:1428] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.069s user 0.008s sys 0.000s
I20260810 23:32:45.433794 13665 tablet_bootstrap.cc:492] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:45.450218 13007 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e26fe4d42445279f12bb17b2841d08 with delete_type TABLET_DATA_DELETED (replaced by tablet b261673faa964b65af3576c89e302864) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:45.454546 13058 raft_consensus.cc:2454] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:45.484526 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f935119e4d9b4accb79db98185c80aca" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:45.485322 13338 raft_consensus.cc:3037] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:45.488876 12223 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 30) took 850 ms (client timeout 1000 ms). Trace:
I20260810 23:32:45.495203 14178 consensus_queue.cc:1035] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:45.495652 13338 raft_consensus.cc:2454] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:45.496778 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_0f0715c81643462985ef73882b0ccb39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:45.499377 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f0715c81643462985ef73882b0ccb39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:45.503976 12698 leader_election.cc:302] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:45.505152 14178 raft_consensus.cc:2781] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:45.506070 14178 raft_consensus.cc:686] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Becoming Leader. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Running, Role: LEADER
I20260810 23:32:45.508337 14178 consensus_queue.cc:237] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.516400 13935 ts_tablet_manager.cc:1918] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:45.520587 13935 log.cc:1196] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/081b4e1f53c74f768a4bf401ae22b585
W20260810 23:32:45.492113 12223 rpcz_store.cc:269] 0810 23:32:44.638509 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:44.703501 (+ 64992us) service_pool.cc:227] Handling call
0810 23:32:45.488774 (+785273us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:45.308854 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:45.309399 (+ 545us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:45.309438 (+ 39us) write_op.cc:435] Acquired schema lock
0810 23:32:45.309457 (+ 19us) tablet.cc:660] Decoding operations
0810 23:32:45.309838 (+ 381us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:45.309888 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:45.309915 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:45.310137 (+ 222us) tablet.cc:699] Row locks acquired
0810 23:32:45.310160 (+ 23us) write_op.cc:260] PREPARE: finished
0810 23:32:45.310313 (+ 153us) write_op.cc:270] Start()
0810 23:32:45.310431 (+ 118us) write_op.cc:276] Timestamp: P: 1786404765310253 usec, L: 0
0810 23:32:45.310452 (+ 21us) op_driver.cc:331] REPLICATION: starting
0810 23:32:45.311215 (+ 763us) log.cc:842] Serialized 536 byte log entry
0810 23:32:45.375693 (+ 64478us) op_driver.cc:444] REPLICATION: finished
0810 23:32:45.376645 (+ 952us) write_op.cc:301] APPLY: starting
0810 23:32:45.376753 (+ 108us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:45.376872 (+ 119us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:45.377005 (+ 133us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:45.377333 (+ 328us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:45.377527 (+ 194us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:45.377577 (+ 50us) write_op.cc:312] APPLY: finished
0810 23:32:45.377828 (+ 251us) log.cc:842] Serialized 23 byte log entry
0810 23:32:45.378013 (+ 185us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:45.378086 (+ 73us) write_op.cc:454] Released schema lock
0810 23:32:45.378122 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:45.378401 (+ 279us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:45.378429 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":14,"child_traces":[["op",{"apply.queue_time_us":469,"apply.run_cpu_time_us":2242,"apply.run_wall_time_us":2242,"num_ops":1,"prepare.queue_time_us":135494,"prepare.run_cpu_time_us":2846,"prepare.run_wall_time_us":23453,"raft.queue_time_us":231,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":17,"replication_time_us":65095}]]}
I20260810 23:32:45.539038 14236 raft_consensus.cc:482] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:45.539448 13935 ts_tablet_manager.cc:1939] T 081b4e1f53c74f768a4bf401ae22b585 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:45.539546 14236 raft_consensus.cc:504] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.542297 14236 leader_election.cc:288] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:45.544945 13665 tablet_bootstrap.cc:654] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:45.558881 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 138) took 4584 ms. Trace:
I20260810 23:32:45.538769 12237 master_service.cc:946] ReplaceTablet: received request to replace tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:45.561941 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081b4e1f53c74f768a4bf401ae22b585" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:45.562857 13338 raft_consensus.cc:2454] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
W20260810 23:32:45.563710 12958 leader_election.cc:341] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 081b4e1f53c74f768a4bf401ae22b585
I20260810 23:32:45.551887 13974 ts_tablet_manager.cc:1918] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:45.564473 12952 leader_election.cc:302] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:45.564986 13974 log.cc:1196] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/b8986e59b2314bd69b9b1de36f1f2ba7
I20260810 23:32:45.565572 14236 raft_consensus.cc:2781] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:45.565980 14236 raft_consensus.cc:482] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:45.566332 14236 raft_consensus.cc:3037] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:45.570276 13974 ts_tablet_manager.cc:1939] T b8986e59b2314bd69b9b1de36f1f2ba7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:45.571209 13514 tablet_bootstrap.cc:492] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:45.571764 13514 ts_tablet_manager.cc:1397] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.443s user 0.012s sys 0.001s
I20260810 23:32:45.574719 13514 raft_consensus.cc:348] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:45.575510 13514 raft_consensus.cc:374] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:45.575788 13514 raft_consensus.cc:729] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:45.578768 13514 consensus_queue.cc:260] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:45.576431 14236 raft_consensus.cc:504] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.581123 14121 ts_tablet_manager.cc:1918] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:45.595775 14121 log.cc:1196] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/79da9583d3524347a9be09e35affaaad
I20260810 23:32:45.597283 12224 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_7331598ab7f64db0bcade7763c41f59f" } new_table_name: "default.Table_b8310838312249ab80ce8a196b1c354f" modify_external_catalogs: true
I20260810 23:32:45.598369 14121 ts_tablet_manager.cc:1939] T 79da9583d3524347a9be09e35affaaad P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:45.600641 13514 ts_tablet_manager.cc:1428] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.028s user 0.009s sys 0.000s
I20260810 23:32:45.600694 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 117) took 5068 ms. Trace:
W20260810 23:32:45.609201 12698 consensus_peers.cc:487] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:45.609364 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081b4e1f53c74f768a4bf401ae22b585" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:45.610056 13338 raft_consensus.cc:3037] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:45.559285 13935 rpcz_store.cc:276] 0810 23:32:40.974590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.003577 (+ 28987us) service_pool.cc:227] Handling call
0810 23:32:45.024980 (+4021403us) ts_tablet_manager.cc:1145] Deleting tablet 081b4e1f53c74f768a4bf401ae22b585
0810 23:32:45.025092 (+ 112us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:45.330281 (+305189us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.488703 (+158422us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.558781 (+ 70078us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4019850}
I20260810 23:32:45.612649 12229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet fd4f55ef709d473f89a16d32160eb61b
I20260810 23:32:45.613070 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 081b4e1f53c74f768a4bf401ae22b585 (table default.Table_fd94688e420b45c2a7f44c6b7db5d7e5 [id=cc72e0bbd749430abf4dabf1d737c92d]) successfully deleted
I20260810 23:32:45.613271 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 131) took 4961 ms. Trace:
I20260810 23:32:45.614169 13935 tablet_replica.cc:330] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:45.613682 13974 rpcz_store.cc:276] 0810 23:32:40.651076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.656761 (+ 5685us) service_pool.cc:227] Handling call
0810 23:32:45.093910 (+4437149us) ts_tablet_manager.cc:1145] Deleting tablet b8986e59b2314bd69b9b1de36f1f2ba7
0810 23:32:45.102441 (+ 8531us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:45.436976 (+334535us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.551543 (+114567us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.611909 (+ 60366us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4436331}
I20260810 23:32:45.615267 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 128) took 4966 ms. Trace:
I20260810 23:32:45.615533 13935 raft_consensus.cc:2229] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:45.622277 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4f55ef709d473f89a16d32160eb61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:45.623735 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4f55ef709d473f89a16d32160eb61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:45.623893 13338 raft_consensus.cc:2454] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:45.628298 13665 tablet_bootstrap.cc:492] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:45.628103 12229 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_f577b98edfc0492cb8edbc6198d6bcd1" } new_table_name: "default.Table_1476071cacef4a1b8681d049b2d23db5" modify_external_catalogs: true
I20260810 23:32:45.628823 13665 ts_tablet_manager.cc:1397] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.195s user 0.015s sys 0.000s
I20260810 23:32:45.629076 13514 tablet_bootstrap.cc:492] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:45.637569 12223 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_cc87f3e94dc84a44822a6f3af8f0c1c3" } modify_external_catalogs: true
I20260810 23:32:45.638569 13935 raft_consensus.cc:2258] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:45.638590 12223 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_cc87f3e94dc84a44822a6f3af8f0c1c3" } modify_external_catalogs: true
W20260810 23:32:45.611598 12958 leader_election.cc:341] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 081b4e1f53c74f768a4bf401ae22b585
I20260810 23:32:45.622375 13007 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4f55ef709d473f89a16d32160eb61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:45.658787 13935 ts_tablet_manager.cc:1905] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:45.638159 13691 tablet_bootstrap.cc:492] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:45.664064 13691 ts_tablet_manager.cc:1397] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.356s user 0.018s sys 0.000s
I20260810 23:32:45.615631 13974 rpcz_store.cc:276] 0810 23:32:40.649083 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.664373 (+ 15290us) service_pool.cc:227] Handling call
0810 23:32:45.614848 (+4950475us) ts_tablet_manager.cc:1145] Deleting tablet 8b1f7c16b6bc4dc0bac80ac55332f056
0810 23:32:45.614927 (+ 79us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:45.615200 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4949344}
W20260810 23:32:45.665537 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 because the tablet was not found. No further retry: Not found: Tablet not found: 8b1f7c16b6bc4dc0bac80ac55332f056
I20260810 23:32:45.666726 13974 tablet_replica.cc:330] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:45.668429 12952 leader_election.cc:302] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
W20260810 23:32:45.669251 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 44177.7us
I20260810 23:32:45.669961 14319 raft_consensus.cc:2781] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:45.670464 14319 raft_consensus.cc:686] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:45.671391 14319 consensus_queue.cc:237] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:45.680727 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 69177us
I20260810 23:32:45.632517 14236 leader_election.cc:288] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:45.667259 13691 raft_consensus.cc:348] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:45.687530 14121 rpcz_store.cc:276] 0810 23:32:40.533043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.542497 (+ 9454us) service_pool.cc:227] Handling call
0810 23:32:45.257755 (+4715258us) ts_tablet_manager.cc:1145] Deleting tablet 79da9583d3524347a9be09e35affaaad
0810 23:32:45.257813 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:45.353674 (+ 95861us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.580755 (+227081us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.600626 (+ 19871us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4714516}
I20260810 23:32:45.688376 13691 raft_consensus.cc:374] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:45.698314 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 79da9583d3524347a9be09e35affaaad (table default.Table_275ea19d22624a12a6b3b2c8e985c4af [id=5b4476e15a7e410599b70f84ab267de9]) successfully deleted
I20260810 23:32:45.631412 13665 raft_consensus.cc:348] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.701267 13514 tablet_bootstrap.cc:654] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:45.700699 13691 raft_consensus.cc:729] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:45.695272 13974 raft_consensus.cc:2229] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:45.699999 13665 raft_consensus.cc:374] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:45.704527 13665 raft_consensus.cc:729] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:45.705428 13974 raft_consensus.cc:2258] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:45.705915 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 129) took 4906 ms. Trace:
I20260810 23:32:45.708062 14121 rpcz_store.cc:276] 0810 23:32:40.799814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.800065 (+ 251us) service_pool.cc:227] Handling call
0810 23:32:45.700266 (+4900201us) ts_tablet_manager.cc:1145] Deleting tablet 42517f11422b42c39db282e2c5f584a3
0810 23:32:45.705433 (+ 5167us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:45.705845 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4899227}
W20260810 23:32:45.711086 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 42517f11422b42c39db282e2c5f584a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42517f11422b42c39db282e2c5f584a3 already in progress: creating tablet
I20260810 23:32:45.709239 13665 consensus_queue.cc:260] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.713490 14121 tablet_replica.cc:330] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:45.717011 13665 ts_tablet_manager.cc:1428] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.088s user 0.007s sys 0.000s
I20260810 23:32:45.719681 14121 raft_consensus.cc:2229] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:45.729238 14121 raft_consensus.cc:2258] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:45.733295 13665 tablet_bootstrap.cc:492] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:45.706087 13691 consensus_queue.cc:260] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:45.739389 13691 ts_tablet_manager.cc:1428] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.075s user 0.007s sys 0.000s
I20260810 23:32:45.741772 13691 tablet_bootstrap.cc:492] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:45.728701 13974 ts_tablet_manager.cc:1905] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:45.762440 12224 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32fad37a28e14903b8bde5aac6ee4e69 has no consensus state
I20260810 23:32:45.768697 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_2ad7503d6e154e87a585df0b30bcbd1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:45.771296 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ad7503d6e154e87a585df0b30bcbd1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:45.735932 14121 ts_tablet_manager.cc:1905] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:45.776794 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_bec275e3dbf54eb08d1df8d68e06b5a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:45.778561 13514 tablet_bootstrap.cc:492] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
W20260810 23:32:45.778738 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bec275e3dbf54eb08d1df8d68e06b5a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:45.779220 13514 ts_tablet_manager.cc:1397] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.177s user 0.012s sys 0.003s
I20260810 23:32:45.789068 13691 tablet_bootstrap.cc:654] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:45.796478 13514 raft_consensus.cc:348] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:45.803946 13514 raft_consensus.cc:374] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:45.816155 13514 raft_consensus.cc:729] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:45.817330 13514 consensus_queue.cc:260] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:45.825338 12215 consensus_peers.cc:487] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 -> Peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Couldn't send request to peer 198f534261994047a68e1d509bb2fd8b. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260810 23:32:45.831463 13514 ts_tablet_manager.cc:1428] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.052s user 0.003s sys 0.003s
I20260810 23:32:45.839600 13665 tablet_bootstrap.cc:654] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:45.857586 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_3afb998b53b2444d831fcfc512a9c665"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:45.859263 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3afb998b53b2444d831fcfc512a9c665 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:45.874337 13935 ts_tablet_manager.cc:1918] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:45.878137 13514 tablet_bootstrap.cc:492] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:45.892234 13935 log.cc:1196] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/42517f11422b42c39db282e2c5f584a3
I20260810 23:32:45.895748 13691 tablet_bootstrap.cc:492] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:45.896288 13691 ts_tablet_manager.cc:1397] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.155s user 0.014s sys 0.002s
I20260810 23:32:45.896855 13935 ts_tablet_manager.cc:1939] T 42517f11422b42c39db282e2c5f584a3 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:45.903012 13514 tablet_bootstrap.cc:654] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:45.900568 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 141) took 4857 ms. Trace:
I20260810 23:32:45.911624 13691 raft_consensus.cc:348] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:45.912289 13935 rpcz_store.cc:276] 0810 23:32:41.043299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.050990 (+ 7691us) service_pool.cc:227] Handling call
0810 23:32:45.613956 (+4562966us) ts_tablet_manager.cc:1145] Deleting tablet 42517f11422b42c39db282e2c5f584a3
0810 23:32:45.614023 (+ 67us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:45.753966 (+139943us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.874026 (+120060us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.900494 (+ 26468us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4545471}
I20260810 23:32:45.914019 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 42517f11422b42c39db282e2c5f584a3 (table default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f]) successfully deleted
I20260810 23:32:45.915181 13935 tablet_replica.cc:330] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:45.916138 13935 raft_consensus.cc:2229] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:45.917217 13935 pending_rounds.cc:62] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3: Trying to abort 1 pending ops.
I20260810 23:32:45.917258 13691 raft_consensus.cc:374] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:45.917088 12226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_7ba7fd44e98542e78c604739a5086a91" } modify_external_catalogs: true
I20260810 23:32:45.917582 13691 raft_consensus.cc:729] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:45.917712 12226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_7ba7fd44e98542e78c604739a5086a91" } modify_external_catalogs: true
I20260810 23:32:45.917663 13935 pending_rounds.cc:68] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317113917569929216 op_type: NO_OP noop_request { }
I20260810 23:32:45.918367 13935 raft_consensus.cc:2866] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:32:45.919481 13935 raft_consensus.cc:2258] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:45.922480 13935 ts_tablet_manager.cc:1905] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:45.926019 13691 consensus_queue.cc:260] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:45.929543 13514 tablet_bootstrap.cc:492] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:45.935874 13514 ts_tablet_manager.cc:1397] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.058s user 0.012s sys 0.001s
I20260810 23:32:45.936398 13974 ts_tablet_manager.cc:1918] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:45.936900 13974 log.cc:1196] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/fd110664bac34f52bfe922a3a76c6290
I20260810 23:32:45.937741 13974 ts_tablet_manager.cc:1939] T fd110664bac34f52bfe922a3a76c6290 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:45.938709 14319 raft_consensus.cc:482] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:45.940140 14319 raft_consensus.cc:504] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:45.939429 13514 raft_consensus.cc:348] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:45.945071 13514 raft_consensus.cc:374] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:45.947635 14319 leader_election.cc:288] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:45.973033 13691 ts_tablet_manager.cc:1428] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.075s user 0.006s sys 0.000s
I20260810 23:32:45.948226 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3a834d77474c3d8feaa5a01f3faff3" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:45.976239 12805 raft_consensus.cc:2454] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:45.979050 12958 leader_election.cc:302] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:45.980789 13691 tablet_bootstrap.cc:492] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:45.980944 14427 raft_consensus.cc:2781] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:45.981276 14427 raft_consensus.cc:482] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:45.981583 14427 raft_consensus.cc:3037] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:45.988471 14121 ts_tablet_manager.cc:1918] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:45.989002 14121 log.cc:1196] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/124259164adf404c9cbe5f4e0d5e16c0
I20260810 23:32:45.989856 14121 ts_tablet_manager.cc:1939] T 124259164adf404c9cbe5f4e0d5e16c0 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:45.994539 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3a834d77474c3d8feaa5a01f3faff3" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:45.997392 12952 leader_election.cc:341] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:46.002633 13514 raft_consensus.cc:729] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:46.012621 14178 raft_consensus.cc:482] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:46.013533 14178 raft_consensus.cc:504] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.017064 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 130) took 5025 ms. Trace:
I20260810 23:32:46.017544 14121 rpcz_store.cc:276] 0810 23:32:40.991426 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.991608 (+ 182us) service_pool.cc:227] Handling call
0810 23:32:45.713298 (+4721690us) ts_tablet_manager.cc:1145] Deleting tablet 124259164adf404c9cbe5f4e0d5e16c0
0810 23:32:45.713350 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:45.876597 (+163247us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.935217 (+ 58620us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:46.016981 (+ 81764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4720748}
I20260810 23:32:46.019167 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 124259164adf404c9cbe5f4e0d5e16c0 (table default.Table_994c5ce0643d422595ad55d7f893ae9d [id=69a8ac9bc8ae4c53aefbed949cf8ac95]) successfully deleted
I20260810 23:32:46.025712 13665 tablet_bootstrap.cc:492] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:46.026207 13665 ts_tablet_manager.cc:1397] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.293s user 0.010s sys 0.009s
I20260810 23:32:46.030467 14178 leader_election.cc:288] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:46.036234 13007 tablet_service.cc:1467] Processing CreateTablet for tablet 422b25457d1549e186641fcedd8feab9 (DEFAULT_TABLE table=default.Table_3f5d5fb567214af3bcd7557741d52620 [id=51cbefc808e244819a59d344cb37c02e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.038158 13007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422b25457d1549e186641fcedd8feab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.037940 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ad5d63a2104995990a86a000963cdc" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:46.039408 13014 tablet_service.cc:1467] Processing CreateTablet for tablet ba8d56858471421a9fbc361ee6aff28f (DEFAULT_TABLE table=default.Table_04d3b5cee61447489f042ab191392be5 [id=9d36464c8de043e7a91673f011908f50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.041252 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8d56858471421a9fbc361ee6aff28f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.034655 12761 tablet_service.cc:1467] Processing CreateTablet for tablet 422b25457d1549e186641fcedd8feab9 (DEFAULT_TABLE table=default.Table_3f5d5fb567214af3bcd7557741d52620 [id=51cbefc808e244819a59d344cb37c02e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:46.040149 12700 leader_election.cc:341] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:46.011660 13514 consensus_queue.cc:260] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:46.048862 12235 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 49) took 1381 ms (client timeout 1000 ms). Trace:
W20260810 23:32:46.049469 12235 rpcz_store.cc:269] 0810 23:32:44.667550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:44.932852 (+265302us) service_pool.cc:227] Handling call
0810 23:32:44.985517 (+ 52665us) spinlock_profiling.cc:243] Waited 8.61 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:46.048759 (+1063242us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":19810176}
I20260810 23:32:46.051416 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422b25457d1549e186641fcedd8feab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.052417 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_c4f98a2612f6454fb7ca19de9ba29ab3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:46.054040 13017 tablet_service.cc:1467] Processing CreateTablet for tablet caee6f3203e049568c4c7cd25684c015 (DEFAULT_TABLE table=default.Table_74329e5e7599420e83c0c8646d466e5b [id=5fff4f293aaa42719e4cd0b3403b59ed]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:46.055130 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4f98a2612f6454fb7ca19de9ba29ab3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:46.055768 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caee6f3203e049568c4c7cd25684c015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.062093 12764 tablet_service.cc:1467] Processing CreateTablet for tablet caee6f3203e049568c4c7cd25684c015 (DEFAULT_TABLE table=default.Table_74329e5e7599420e83c0c8646d466e5b [id=5fff4f293aaa42719e4cd0b3403b59ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.064258 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caee6f3203e049568c4c7cd25684c015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.066998 12769 tablet_service.cc:1467] Processing CreateTablet for tablet bd921e48f22348639f7549d47b165185 (DEFAULT_TABLE table=default.Table_9a2539ffd1684a4cae73196a0ab97172 [id=254e99048db04a75ae17a9f262c8a0c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.059579 12763 tablet_service.cc:1467] Processing CreateTablet for tablet ba8d56858471421a9fbc361ee6aff28f (DEFAULT_TABLE table=default.Table_04d3b5cee61447489f042ab191392be5 [id=9d36464c8de043e7a91673f011908f50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.068710 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd921e48f22348639f7549d47b165185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.069658 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8d56858471421a9fbc361ee6aff28f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.074651 13514 ts_tablet_manager.cc:1428] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.138s user 0.007s sys 0.000s
I20260810 23:32:46.084992 13514 tablet_bootstrap.cc:492] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:32:46.089252 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet e62dc27b59ba469d99e8d11372aceff5 (table default.Table_3b251f8589be43f29704b8906d49f6cd [id=c56022cc8cad4c7cbefafad0782cefde], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e62dc27b59ba469d99e8d11372aceff5 consensus state has no leader
I20260810 23:32:46.102828 13020 tablet_service.cc:1467] Processing CreateTablet for tablet bd921e48f22348639f7549d47b165185 (DEFAULT_TABLE table=default.Table_9a2539ffd1684a4cae73196a0ab97172 [id=254e99048db04a75ae17a9f262c8a0c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.105536 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd921e48f22348639f7549d47b165185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.130596 12223 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 80b54edbb45d496c94efedb6feee8e45
I20260810 23:32:46.133008 12224 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_65718901959a4edaaa7370e442209578" } new_table_name: "default.Table_1ec6152b7f994da6987cb0683216ade9" modify_external_catalogs: true
I20260810 23:32:46.140821 13514 tablet_bootstrap.cc:654] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:46.141621 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_c5751d9b450d4231a2d59983159349b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:46.144289 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5751d9b450d4231a2d59983159349b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:46.146397 12756 tablet_service.cc:1467] Processing CreateTablet for tablet a6f53aca31ee4e338d9b5e281557c121 (DEFAULT_TABLE table=default.Table_d860efaf3e3a4d1386d4c8e328463a8a [id=89c842ab64c546bbb967a2246a7220ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.142508 12766 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b54edbb45d496c94efedb6feee8e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:46.149614 13008 tablet_service.cc:1467] Processing CreateTablet for tablet a6f53aca31ee4e338d9b5e281557c121 (DEFAULT_TABLE table=default.Table_d860efaf3e3a4d1386d4c8e328463a8a [id=89c842ab64c546bbb967a2246a7220ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.155313 13665 raft_consensus.cc:348] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:46.157295 13665 raft_consensus.cc:374] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:46.157351 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 131) took 5027 ms. Trace:
I20260810 23:32:46.157647 13665 raft_consensus.cc:729] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:46.160425 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 129) took 5511 ms. Trace:
I20260810 23:32:46.160614 13274 tablet_service.cc:1467] Processing CreateTablet for tablet bd921e48f22348639f7549d47b165185 (DEFAULT_TABLE table=default.Table_9a2539ffd1684a4cae73196a0ab97172 [id=254e99048db04a75ae17a9f262c8a0c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.160893 13974 rpcz_store.cc:276] 0810 23:32:40.649472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:40.665691 (+ 16219us) service_pool.cc:227] Handling call
0810 23:32:45.666516 (+5000825us) ts_tablet_manager.cc:1145] Deleting tablet fd110664bac34f52bfe922a3a76c6290
0810 23:32:45.666583 (+ 67us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:45.824610 (+158027us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:45.936092 (+111482us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:46.160337 (+224245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4999878}
I20260810 23:32:46.155323 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ad5d63a2104995990a86a000963cdc" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:46.162293 13274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd921e48f22348639f7549d47b165185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.162655 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 141) took 3381 ms. Trace:
I20260810 23:32:46.163009 13974 rpcz_store.cc:276] 0810 23:32:42.781873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.784368 (+ 2495us) service_pool.cc:227] Handling call
0810 23:32:46.162168 (+3377800us) ts_tablet_manager.cc:1145] Deleting tablet 124259164adf404c9cbe5f4e0d5e16c0
0810 23:32:46.162226 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.162595 (+ 369us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3375555}
I20260810 23:32:46.164327 13665 consensus_queue.cc:260] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:46.164731 12698 leader_election.cc:341] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:46.165601 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b54edbb45d496c94efedb6feee8e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:46.165531 12698 leader_election.cc:302] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:46.167912 14278 raft_consensus.cc:2726] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:46.171043 14121 rpcz_store.cc:276] 0810 23:32:41.130357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.130543 (+ 186us) service_pool.cc:227] Handling call
0810 23:32:46.156913 (+5026370us) ts_tablet_manager.cc:1145] Deleting tablet ca0dd19112504f8ca57f64dcf0bd50a8
0810 23:32:46.156971 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.157279 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5025571}
I20260810 23:32:46.172397 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 132) took 5027 ms. Trace:
I20260810 23:32:46.172711 14121 rpcz_store.cc:276] 0810 23:32:41.144611 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.163213 (+ 18602us) service_pool.cc:227] Handling call
0810 23:32:46.171983 (+5008770us) ts_tablet_manager.cc:1145] Deleting tablet 133654008f5247b2ac87dcd09195f197
0810 23:32:46.172054 (+ 71us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.172341 (+ 287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5004234}
W20260810 23:32:46.173861 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet ca0dd19112504f8ca57f64dcf0bd50a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca0dd19112504f8ca57f64dcf0bd50a8 already in progress: creating tablet
W20260810 23:32:46.174852 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 133654008f5247b2ac87dcd09195f197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 133654008f5247b2ac87dcd09195f197 already in progress: creating tablet
I20260810 23:32:46.175434 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 80b54edbb45d496c94efedb6feee8e45 (DEFAULT_TABLE table=default.Table_cc87f3e94dc84a44822a6f3af8f0c1c3 [id=2141df5169c64180b39b058b342f779d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.151561 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f53aca31ee4e338d9b5e281557c121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.176143 12766 tablet_service.cc:1467] Processing CreateTablet for tablet 80b54edbb45d496c94efedb6feee8e45 (DEFAULT_TABLE table=default.Table_cc87f3e94dc84a44822a6f3af8f0c1c3 [id=2141df5169c64180b39b058b342f779d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.033711 14427 raft_consensus.cc:504] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.166118 13008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f53aca31ee4e338d9b5e281557c121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.178692 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b54edbb45d496c94efedb6feee8e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.181805 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b54edbb45d496c94efedb6feee8e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.196913 13007 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b54edbb45d496c94efedb6feee8e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:46.214620 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 422b25457d1549e186641fcedd8feab9 (DEFAULT_TABLE table=default.Table_3f5d5fb567214af3bcd7557741d52620 [id=51cbefc808e244819a59d344cb37c02e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.232604 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet fd110664bac34f52bfe922a3a76c6290 (table default.Table_0af7ca97e51c47a29d5dd8867b152066 [id=32144cd75df84cfabce64657802c111a]) successfully deleted
W20260810 23:32:46.233678 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 124259164adf404c9cbe5f4e0d5e16c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 124259164adf404c9cbe5f4e0d5e16c0 already in progress: creating tablet
I20260810 23:32:46.244868 14427 leader_election.cc:288] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:46.246527 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3a834d77474c3d8feaa5a01f3faff3" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:46.247670 12805 raft_consensus.cc:3037] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:46.217720 13278 tablet_service.cc:1467] Processing CreateTablet for tablet ba8d56858471421a9fbc361ee6aff28f (DEFAULT_TABLE table=default.Table_04d3b5cee61447489f042ab191392be5 [id=9d36464c8de043e7a91673f011908f50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.220669 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3a834d77474c3d8feaa5a01f3faff3" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:46.247241 12761 tablet_service.cc:1467] Processing CreateTablet for tablet e62dc27b59ba469d99e8d11372aceff5 (DEFAULT_TABLE table=default.Table_3b251f8589be43f29704b8906d49f6cd [id=c56022cc8cad4c7cbefafad0782cefde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.223551 13276 tablet_service.cc:1467] Processing CreateTablet for tablet a6f53aca31ee4e338d9b5e281557c121 (DEFAULT_TABLE table=default.Table_d860efaf3e3a4d1386d4c8e328463a8a [id=89c842ab64c546bbb967a2246a7220ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.254771 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62dc27b59ba469d99e8d11372aceff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.220374 13273 tablet_service.cc:1467] Processing CreateTablet for tablet caee6f3203e049568c4c7cd25684c015 (DEFAULT_TABLE table=default.Table_74329e5e7599420e83c0c8646d466e5b [id=5fff4f293aaa42719e4cd0b3403b59ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.227417 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 140) took 3446 ms. Trace:
I20260810 23:32:46.260274 13974 rpcz_store.cc:276] 0810 23:32:42.781147 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.788397 (+ 7250us) service_pool.cc:227] Handling call
0810 23:32:46.163999 (+3375602us) ts_tablet_manager.cc:1145] Deleting tablet 42517f11422b42c39db282e2c5f584a3
0810 23:32:46.172445 (+ 8446us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.227322 (+ 54877us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3373416}
I20260810 23:32:46.240953 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 80b54edbb45d496c94efedb6feee8e45 (DEFAULT_TABLE table=default.Table_cc87f3e94dc84a44822a6f3af8f0c1c3 [id=2141df5169c64180b39b058b342f779d]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:46.265014 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 42517f11422b42c39db282e2c5f584a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42517f11422b42c39db282e2c5f584a3 already in progress: creating tablet
I20260810 23:32:46.246106 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 134) took 4712 ms. Trace:
W20260810 23:32:46.264539 12952 leader_election.cc:341] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:46.277947 13691 tablet_bootstrap.cc:654] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:46.279871 14307 raft_consensus.cc:482] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:46.280977 14307 raft_consensus.cc:504] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:46.287972 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 39ff3bb1947c4f689353587396d878b8 (DEFAULT_TABLE table=default.Table_e1e8440f714f4d55a1622ec3b2f537b6 [id=f7cc6d155f6a451e9f7109d7be8d8f83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.290299 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ff3bb1947c4f689353587396d878b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.277580 14121 rpcz_store.cc:276] 0810 23:32:41.534158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.534458 (+ 300us) service_pool.cc:227] Handling call
0810 23:32:46.245675 (+4711217us) ts_tablet_manager.cc:1145] Deleting tablet a41ddf9f6f71443ba75c97d2b010372c
0810 23:32:46.245734 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.246034 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4710014}
I20260810 23:32:46.294118 13277 tablet_service.cc:1467] Processing CreateTablet for tablet e62dc27b59ba469d99e8d11372aceff5 (DEFAULT_TABLE table=default.Table_3b251f8589be43f29704b8906d49f6cd [id=c56022cc8cad4c7cbefafad0782cefde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.295902 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 135) took 4709 ms. Trace:
I20260810 23:32:46.297873 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 39ff3bb1947c4f689353587396d878b8 (DEFAULT_TABLE table=default.Table_e1e8440f714f4d55a1622ec3b2f537b6 [id=f7cc6d155f6a451e9f7109d7be8d8f83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.299221 14121 rpcz_store.cc:276] 0810 23:32:41.586922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.587336 (+ 414us) service_pool.cc:227] Handling call
0810 23:32:46.295358 (+4708022us) ts_tablet_manager.cc:1145] Deleting tablet 8b1f7c16b6bc4dc0bac80ac55332f056
0810 23:32:46.295434 (+ 76us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.295821 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4706385}
I20260810 23:32:46.301180 13665 ts_tablet_manager.cc:1428] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.275s user 0.008s sys 0.003s
W20260810 23:32:46.303740 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet a41ddf9f6f71443ba75c97d2b010372c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a41ddf9f6f71443ba75c97d2b010372c already in progress: creating tablet
W20260810 23:32:46.307026 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 8b1f7c16b6bc4dc0bac80ac55332f056 because the tablet was not found. No further retry: Not found: Tablet not found: 8b1f7c16b6bc4dc0bac80ac55332f056
I20260810 23:32:46.306969 14307 leader_election.cc:288] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:46.310328 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b261673faa964b65af3576c89e302864" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:46.310913 14121 tablet_replica.cc:330] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:46.311697 13062 raft_consensus.cc:2379] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in current term 1: Already voted for candidate 191d120f4c7f44fc8475305a6d2e75f3 in this term.
I20260810 23:32:46.317046 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b261673faa964b65af3576c89e302864" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:46.334966 12805 raft_consensus.cc:2454] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:46.312005 14121 raft_consensus.cc:2229] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:46.319290 13213 leader_election.cc:302] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:46.323551 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8d56858471421a9fbc361ee6aff28f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.346215 13273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caee6f3203e049568c4c7cd25684c015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.341920 13007 tablet_service.cc:1467] Processing CreateTablet for tablet 39ff3bb1947c4f689353587396d878b8 (DEFAULT_TABLE table=default.Table_e1e8440f714f4d55a1622ec3b2f537b6 [id=f7cc6d155f6a451e9f7109d7be8d8f83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.341763 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 143) took 3558 ms. Trace:
I20260810 23:32:46.349859 13007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ff3bb1947c4f689353587396d878b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.349335 13974 rpcz_store.cc:276] 0810 23:32:42.783214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.789598 (+ 6384us) service_pool.cc:227] Handling call
0810 23:32:46.262216 (+3472618us) ts_tablet_manager.cc:1145] Deleting tablet ca0dd19112504f8ca57f64dcf0bd50a8
0810 23:32:46.329044 (+ 66828us) spinlock_profiling.cc:243] Waited 53.4 ms on lock 0x7b5800001898. stack: 00007f8764d74033 00007f8764d73f30 00007f87648f5534 00000000004db2b4 00000000004db249 00007f8764beaf11 00007f8764d6e851 00007f8764d6ee82 00007f876dc2e8b9 00007f876cc625d8 00007f876cd26389 00007f876cd26983 00007f876cd56e10 00007f876cd3de0d 00007f876cd3ddb9 00007f876cd3dd49
0810 23:32:46.341201 (+ 12157us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.341675 (+ 474us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":122914048,"tablet-delete.queue_time_us":3470577}
W20260810 23:32:46.353850 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet ca0dd19112504f8ca57f64dcf0bd50a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca0dd19112504f8ca57f64dcf0bd50a8 already in progress: creating tablet
I20260810 23:32:46.355823 14121 raft_consensus.cc:2258] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:46.348443 13277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62dc27b59ba469d99e8d11372aceff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.366439 13279 tablet_service.cc:1467] Processing CreateTablet for tablet 32fad37a28e14903b8bde5aac6ee4e69 (DEFAULT_TABLE table=default.Table_b8310838312249ab80ce8a196b1c354f [id=9290d4418f5a4ca9a7d7768dae5843a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.368055 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fad37a28e14903b8bde5aac6ee4e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.373458 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f53aca31ee4e338d9b5e281557c121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.359669 13020 tablet_service.cc:1467] Processing CreateTablet for tablet e62dc27b59ba469d99e8d11372aceff5 (DEFAULT_TABLE table=default.Table_3b251f8589be43f29704b8906d49f6cd [id=c56022cc8cad4c7cbefafad0782cefde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.371835 13691 tablet_bootstrap.cc:492] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:46.376859 13691 ts_tablet_manager.cc:1397] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.396s user 0.012s sys 0.008s
I20260810 23:32:46.377265 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62dc27b59ba469d99e8d11372aceff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.381203 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b54edbb45d496c94efedb6feee8e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.381999 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ff3bb1947c4f689353587396d878b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.383828 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 144) took 3600 ms. Trace:
I20260810 23:32:46.384449 13974 rpcz_store.cc:276] 0810 23:32:42.783849 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.808166 (+ 24317us) service_pool.cc:227] Handling call
0810 23:32:46.352541 (+3544375us) ts_tablet_manager.cc:1145] Deleting tablet 133654008f5247b2ac87dcd09195f197
0810 23:32:46.352597 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.383751 (+ 31154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3522435}
W20260810 23:32:46.386457 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 133654008f5247b2ac87dcd09195f197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 133654008f5247b2ac87dcd09195f197 already in progress: creating tablet
I20260810 23:32:46.387125 13031 tablet_service.cc:1467] Processing CreateTablet for tablet 32fad37a28e14903b8bde5aac6ee4e69 (DEFAULT_TABLE table=default.Table_b8310838312249ab80ce8a196b1c354f [id=9290d4418f5a4ca9a7d7768dae5843a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.388423 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fad37a28e14903b8bde5aac6ee4e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.390856 12958 leader_election.cc:302] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:46.392679 14427 raft_consensus.cc:2781] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:46.397435 14427 raft_consensus.cc:686] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:46.398609 14427 consensus_queue.cc:237] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.407938 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422b25457d1549e186641fcedd8feab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.338022 13935 ts_tablet_manager.cc:1918] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:46.361034 12766 tablet_service.cc:1467] Processing CreateTablet for tablet 32fad37a28e14903b8bde5aac6ee4e69 (DEFAULT_TABLE table=default.Table_b8310838312249ab80ce8a196b1c354f [id=9290d4418f5a4ca9a7d7768dae5843a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.359951 13514 tablet_bootstrap.cc:492] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:46.384748 14178 consensus_queue.cc:1035] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:46.412117 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fad37a28e14903b8bde5aac6ee4e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.413838 13935 log.cc:1196] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/124259164adf404c9cbe5f4e0d5e16c0
I20260810 23:32:46.414803 13935 ts_tablet_manager.cc:1939] T 124259164adf404c9cbe5f4e0d5e16c0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:46.416440 13665 tablet_bootstrap.cc:492] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:46.422509 14216 raft_consensus.cc:482] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:46.423063 14216 raft_consensus.cc:504] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.428944 14216 leader_election.cc:288] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:46.430182 13514 ts_tablet_manager.cc:1397] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.345s user 0.020s sys 0.006s
I20260810 23:32:46.429924 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db363eb0f98d460eb6f5a577233d8ac0" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:46.431914 12952 leader_election.cc:341] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:46.433758 13514 raft_consensus.cc:348] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.436223 13514 raft_consensus.cc:374] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:46.436266 13691 raft_consensus.cc:348] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.437080 13691 raft_consensus.cc:374] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:46.437348 13691 raft_consensus.cc:729] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:46.437029 13514 raft_consensus.cc:729] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:46.438040 13691 consensus_queue.cc:260] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.440332 13691 ts_tablet_manager.cc:1428] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.063s user 0.007s sys 0.000s
I20260810 23:32:46.441349 13691 tablet_bootstrap.cc:492] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:46.444195 14178 raft_consensus.cc:482] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:46.444754 14178 raft_consensus.cc:504] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.447255 13691 tablet_bootstrap.cc:654] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:46.448082 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b71d565669f40d8a6d06ab267069430" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:46.449826 12700 leader_election.cc:341] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:46.451790 14178 leader_election.cc:288] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:46.451063 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db363eb0f98d460eb6f5a577233d8ac0" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
W20260810 23:32:46.455294 12958 leader_election.cc:341] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:46.461318 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 146) took 3580 ms. Trace:
I20260810 23:32:46.461737 13974 rpcz_store.cc:276] 0810 23:32:42.880908 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.891620 (+ 10712us) service_pool.cc:227] Handling call
0810 23:32:46.385559 (+3493939us) ts_tablet_manager.cc:1145] Deleting tablet a41ddf9f6f71443ba75c97d2b010372c
0810 23:32:46.385612 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.461215 (+ 75603us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3492637}
I20260810 23:32:46.463126 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 148) took 3581 ms. Trace:
I20260810 23:32:46.455812 12958 leader_election.cc:302] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:46.464368 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 142) took 5410 ms. Trace:
I20260810 23:32:46.464694 13935 rpcz_store.cc:276] 0810 23:32:41.053871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.197633 (+143762us) service_pool.cc:227] Handling call
0810 23:32:45.914984 (+4717351us) ts_tablet_manager.cc:1145] Deleting tablet 124259164adf404c9cbe5f4e0d5e16c0
0810 23:32:45.915042 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:45.999280 (+ 84238us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:46.326866 (+327586us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:46.464287 (+137421us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4716296}
W20260810 23:32:46.465166 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet a41ddf9f6f71443ba75c97d2b010372c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a41ddf9f6f71443ba75c97d2b010372c already in progress: creating tablet
I20260810 23:32:46.467603 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 124259164adf404c9cbe5f4e0d5e16c0 (table default.Table_994c5ce0643d422595ad55d7f893ae9d [id=69a8ac9bc8ae4c53aefbed949cf8ac95]) successfully deleted
I20260810 23:32:46.463605 13974 rpcz_store.cc:276] 0810 23:32:42.882070 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.896785 (+ 14715us) service_pool.cc:227] Handling call
0810 23:32:46.462707 (+3565922us) ts_tablet_manager.cc:1145] Deleting tablet dd83bbe7a1664c278098fa66b9c6895e
0810 23:32:46.462770 (+ 63us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.463039 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3564003}
W20260810 23:32:46.475629 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet dd83bbe7a1664c278098fa66b9c6895e because the tablet was not found. No further retry: Not found: Tablet not found: dd83bbe7a1664c278098fa66b9c6895e
W20260810 23:32:46.476946 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 157780us
I20260810 23:32:46.480491 14121 ts_tablet_manager.cc:1905] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:46.483171 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 143) took 5218 ms. Trace:
I20260810 23:32:46.488327 14427 raft_consensus.cc:2726] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:46.492923 13025 tablet_service.cc:1467] Processing CreateTablet for tablet 0be35dabbec44caebd1df687b167f34a (DEFAULT_TABLE table=default.Table_216b21581f454466a84891c7e3d0e567 [id=ff7ae9781f734fe1bd9565c344b6d5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.493443 13935 rpcz_store.cc:276] 0810 23:32:41.264822 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.306155 (+ 41333us) service_pool.cc:227] Handling call
0810 23:32:46.465806 (+5159651us) ts_tablet_manager.cc:1145] Deleting tablet ca0dd19112504f8ca57f64dcf0bd50a8
0810 23:32:46.482304 (+ 16498us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.483075 (+ 771us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5158756}
I20260810 23:32:46.494983 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be35dabbec44caebd1df687b167f34a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:46.494930 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet ca0dd19112504f8ca57f64dcf0bd50a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca0dd19112504f8ca57f64dcf0bd50a8 already in progress: creating tablet
I20260810 23:32:46.500413 13935 tablet_replica.cc:330] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:46.504472 13935 raft_consensus.cc:2229] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:46.505167 13935 raft_consensus.cc:2258] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:46.507603 13935 ts_tablet_manager.cc:1905] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:46.512354 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 149) took 3629 ms. Trace:
I20260810 23:32:46.512778 13974 rpcz_store.cc:276] 0810 23:32:42.882605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.905210 (+ 22605us) service_pool.cc:227] Handling call
0810 23:32:46.476836 (+3571626us) ts_tablet_manager.cc:1145] Deleting tablet 423e8d583d704fa9834026ebcd57e17e
0810 23:32:46.476889 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.512265 (+ 35376us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3570207}
W20260810 23:32:46.524828 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 423e8d583d704fa9834026ebcd57e17e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 423e8d583d704fa9834026ebcd57e17e already in progress: creating tablet
I20260810 23:32:46.512570 14417 raft_consensus.cc:482] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:46.526374 14417 raft_consensus.cc:504] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:46.528849 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 157) took 2398 ms. Trace:
I20260810 23:32:46.529219 13974 rpcz_store.cc:276] 0810 23:32:44.130653 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:44.150511 (+ 19858us) service_pool.cc:227] Handling call
0810 23:32:46.513708 (+2363197us) ts_tablet_manager.cc:1145] Deleting tablet 440de7d9de8d4be9a5322e1fe3388b9b
0810 23:32:46.513758 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.528772 (+ 15014us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2341120}
I20260810 23:32:46.531394 14417 leader_election.cc:288] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:46.531486 13023 tablet_service.cc:1467] Processing CreateTablet for tablet 31730595c84a44579c4e9db09ef603ba (DEFAULT_TABLE table=default.Table_6b8e1ec40cdc4eda823368ca59437332 [id=e2df41b29ef8482b9ad579c172717ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.512898 12761 tablet_service.cc:1467] Processing CreateTablet for tablet 0be35dabbec44caebd1df687b167f34a (DEFAULT_TABLE table=default.Table_216b21581f454466a84891c7e3d0e567 [id=ff7ae9781f734fe1bd9565c344b6d5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.516497 13514 consensus_queue.cc:260] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.516805 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 31730595c84a44579c4e9db09ef603ba (DEFAULT_TABLE table=default.Table_6b8e1ec40cdc4eda823368ca59437332 [id=e2df41b29ef8482b9ad579c172717ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.534449 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be35dabbec44caebd1df687b167f34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.535719 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31730595c84a44579c4e9db09ef603ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.479180 14307 raft_consensus.cc:482] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:46.526364 14476 raft_consensus.cc:3037] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:46.543076 12763 tablet_service.cc:1467] Processing CreateTablet for tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd (DEFAULT_TABLE table=default.Table_9625301c8cda460981e80e753c785d4b [id=d79b564767e24f3b947e21b4e56e2e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.545495 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.560597 13514 ts_tablet_manager.cc:1428] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.130s user 0.008s sys 0.000s
I20260810 23:32:46.564548 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd (DEFAULT_TABLE table=default.Table_9625301c8cda460981e80e753c785d4b [id=d79b564767e24f3b947e21b4e56e2e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.569684 13514 tablet_bootstrap.cc:492] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:46.573300 13691 tablet_bootstrap.cc:492] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:46.573827 13691 ts_tablet_manager.cc:1397] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.133s user 0.016s sys 0.002s
I20260810 23:32:46.542415 14307 raft_consensus.cc:504] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:46.581475 13691 raft_consensus.cc:348] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.582580 13691 raft_consensus.cc:374] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:46.588295 13279 tablet_service.cc:1467] Processing CreateTablet for tablet 0be35dabbec44caebd1df687b167f34a (DEFAULT_TABLE table=default.Table_216b21581f454466a84891c7e3d0e567 [id=ff7ae9781f734fe1bd9565c344b6d5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.592497 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 31730595c84a44579c4e9db09ef603ba (DEFAULT_TABLE table=default.Table_6b8e1ec40cdc4eda823368ca59437332 [id=e2df41b29ef8482b9ad579c172717ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:46.595634 13691 raft_consensus.cc:729] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:46.596699 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 158) took 2444 ms. Trace:
I20260810 23:32:46.597107 13974 rpcz_store.cc:276] 0810 23:32:44.151864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:44.154046 (+ 2182us) service_pool.cc:227] Handling call
0810 23:32:46.530107 (+2376061us) ts_tablet_manager.cc:1145] Deleting tablet 4f3049bcdc6e43c795961c74f0d008f5
0810 23:32:46.596109 (+ 66002us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.596612 (+ 503us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2357232}
I20260810 23:32:46.598785 13691 consensus_queue.cc:260] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.600442 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31730595c84a44579c4e9db09ef603ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.614784 13691 ts_tablet_manager.cc:1428] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.041s user 0.008s sys 0.002s
I20260810 23:32:46.621160 13935 ts_tablet_manager.cc:1918] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:46.623606 13935 log.cc:1196] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/133654008f5247b2ac87dcd09195f197
I20260810 23:32:46.627104 13935 ts_tablet_manager.cc:1939] T 133654008f5247b2ac87dcd09195f197 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:46.636318 14307 leader_election.cc:288] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:46.640933 13665 tablet_bootstrap.cc:654] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:46.648845 13974 tablet_replica.cc:330] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:46.649948 13974 raft_consensus.cc:2229] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:46.650962 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be35dabbec44caebd1df687b167f34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.652196 13514 tablet_bootstrap.cc:654] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:46.653654 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 144) took 5325 ms. Trace:
I20260810 23:32:46.654208 13935 rpcz_store.cc:276] 0810 23:32:41.307600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.307733 (+ 133us) service_pool.cc:227] Handling call
0810 23:32:46.500183 (+5192450us) ts_tablet_manager.cc:1145] Deleting tablet 133654008f5247b2ac87dcd09195f197
0810 23:32:46.500256 (+ 73us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.593104 (+ 92848us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:46.620785 (+ 27681us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:46.632979 (+ 12194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5191474}
I20260810 23:32:46.656749 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 133654008f5247b2ac87dcd09195f197 (table default.Table_d664d07d116e4f2aa4fc263449eb8dd0 [id=04ed6428e4df49b2ab0438035da70731]) successfully deleted
W20260810 23:32:46.656805 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 440de7d9de8d4be9a5322e1fe3388b9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 440de7d9de8d4be9a5322e1fe3388b9b already in progress: creating tablet
I20260810 23:32:46.657142 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd (DEFAULT_TABLE table=default.Table_9625301c8cda460981e80e753c785d4b [id=d79b564767e24f3b947e21b4e56e2e3b]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:46.658036 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 4f3049bcdc6e43c795961c74f0d008f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f3049bcdc6e43c795961c74f0d008f5 already in progress: creating tablet
I20260810 23:32:46.658319 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.659797 13974 raft_consensus.cc:2258] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:46.666945 13974 ts_tablet_manager.cc:1905] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:46.677942 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 146) took 5121 ms. Trace:
I20260810 23:32:46.682556 13935 rpcz_store.cc:276] 0810 23:32:41.556734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.556906 (+ 172us) service_pool.cc:227] Handling call
0810 23:32:46.677335 (+5120429us) ts_tablet_manager.cc:1145] Deleting tablet a41ddf9f6f71443ba75c97d2b010372c
0810 23:32:46.677420 (+ 85us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.677849 (+ 429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5083878}
W20260810 23:32:46.684254 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet a41ddf9f6f71443ba75c97d2b010372c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a41ddf9f6f71443ba75c97d2b010372c already in progress: creating tablet
I20260810 23:32:46.686648 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 147) took 5040 ms. Trace:
I20260810 23:32:46.688659 13935 rpcz_store.cc:276] 0810 23:32:41.646153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.646341 (+ 188us) service_pool.cc:227] Handling call
0810 23:32:46.685640 (+5039299us) ts_tablet_manager.cc:1145] Deleting tablet 423e8d583d704fa9834026ebcd57e17e
0810 23:32:46.685787 (+ 147us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.686404 (+ 617us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4993065}
W20260810 23:32:46.691951 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 423e8d583d704fa9834026ebcd57e17e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 423e8d583d704fa9834026ebcd57e17e already in progress: creating tablet
I20260810 23:32:46.693821 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 149) took 4692 ms. Trace:
I20260810 23:32:46.694650 13935 rpcz_store.cc:276] 0810 23:32:42.001438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:42.001587 (+ 149us) service_pool.cc:227] Handling call
0810 23:32:46.693131 (+4691544us) ts_tablet_manager.cc:1145] Deleting tablet 3d596299c82b4ff1ac151d68b97ff55b
0810 23:32:46.693248 (+ 117us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.693718 (+ 470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4667971}
I20260810 23:32:46.698822 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 148) took 4723 ms. Trace:
I20260810 23:32:46.702097 13514 tablet_bootstrap.cc:492] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:46.702754 13514 ts_tablet_manager.cc:1397] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.133s user 0.011s sys 0.007s
I20260810 23:32:46.706605 13514 raft_consensus.cc:348] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:46.696552 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 3d596299c82b4ff1ac151d68b97ff55b because the tablet was not found. No further retry: Not found: Tablet not found: 3d596299c82b4ff1ac151d68b97ff55b
I20260810 23:32:46.713321 14427 raft_consensus.cc:482] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:46.713994 14427 raft_consensus.cc:504] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.699473 13935 rpcz_store.cc:276] 0810 23:32:41.976010 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.998594 (+ 22584us) service_pool.cc:227] Handling call
0810 23:32:46.698297 (+4699703us) ts_tablet_manager.cc:1145] Deleting tablet dd83bbe7a1664c278098fa66b9c6895e
0810 23:32:46.698391 (+ 94us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.698746 (+ 355us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4634068}
I20260810 23:32:46.717356 13935 tablet_replica.cc:330] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:46.718406 13514 raft_consensus.cc:374] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:46.718704 13514 raft_consensus.cc:729] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:46.718603 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440de7d9de8d4be9a5322e1fe3388b9b" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:46.719344 12805 raft_consensus.cc:2454] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:46.719507 13514 consensus_queue.cc:260] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:46.721297 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet dd83bbe7a1664c278098fa66b9c6895e because the tablet was not found. No further retry: Not found: Tablet not found: dd83bbe7a1664c278098fa66b9c6895e
I20260810 23:32:46.723484 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440de7d9de8d4be9a5322e1fe3388b9b" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:46.725620 12958 leader_election.cc:302] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:46.727880 14216 raft_consensus.cc:2781] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:46.728277 14216 raft_consensus.cc:482] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:46.728456 13514 ts_tablet_manager.cc:1428] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
I20260810 23:32:46.728567 14216 raft_consensus.cc:3037] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:46.729517 13514 tablet_bootstrap.cc:492] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:32:46.733300 12952 leader_election.cc:341] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:46.733713 14476 raft_consensus.cc:2726] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:46.735108 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b71d565669f40d8a6d06ab267069430" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:46.736162 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d697e7533af4fc0a15cb41f3fcce4db" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:46.741123 13331 raft_consensus.cc:3037] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:46.747377 13665 tablet_bootstrap.cc:492] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:46.747875 14427 leader_election.cc:288] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:46.758322 14216 raft_consensus.cc:504] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.748351 13665 ts_tablet_manager.cc:1397] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.332s user 0.009s sys 0.009s
W20260810 23:32:46.780992 12698 leader_election.cc:341] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:46.781777 12698 leader_election.cc:302] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:32:46.784904 13014 consensus_meta.cc:287] T ba8d56858471421a9fbc361ee6aff28f P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.513s user 0.013s sys 0.000s
I20260810 23:32:46.788059 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d697e7533af4fc0a15cb41f3fcce4db" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:46.788069 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd77526f1db4455291fde8a001092a09" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:46.797197 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31730595c84a44579c4e9db09ef603ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.798116 13691 tablet_bootstrap.cc:492] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:46.811548 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:46.830284 14476 consensus_queue.cc:1035] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:32:46.836891 12700 leader_election.cc:341] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:46.838719 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd77526f1db4455291fde8a001092a09" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:46.840121 13691 tablet_bootstrap.cc:654] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:46.859894 13514 tablet_bootstrap.cc:654] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:46.869714 14417 raft_consensus.cc:2726] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:46.878322 14278 consensus_queue.cc:1035] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:46.885268 13691 tablet_bootstrap.cc:492] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:46.885854 13691 ts_tablet_manager.cc:1397] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.088s user 0.017s sys 0.006s
I20260810 23:32:46.886032 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440de7d9de8d4be9a5322e1fe3388b9b" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
W20260810 23:32:46.888726 12952 leader_election.cc:341] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:46.890487 13691 raft_consensus.cc:348] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:46.891390 13691 raft_consensus.cc:374] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:46.891549 14216 leader_election.cc:288] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:46.892966 13935 raft_consensus.cc:2229] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Raft consensus shutting down.
W20260810 23:32:46.895658 13213 consensus_peers.cc:487] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 840400119d20424ea0f03f2bff550d44. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:46.900920 13691 raft_consensus.cc:729] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:46.902002 13691 consensus_queue.cc:260] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:46.905233 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 124368us
W20260810 23:32:46.906603 12958 consensus_peers.cc:487] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 081b4e1f53c74f768a4bf401ae22b585. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:46.906127 13008 consensus_meta.cc:287] T a6f53aca31ee4e338d9b5e281557c121 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.586s user 0.006s sys 0.007s
I20260810 23:32:46.931156 12240 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet bd77526f1db4455291fde8a001092a09 (table default.Table_ac23e0995fcd414ead1b88406cf7d79e [id=963512beef0d4063b8e6a5738abd1d92]). Expected version 2 got 0
I20260810 23:32:46.932710 12240 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
W20260810 23:32:46.933063 13031 consensus_meta.cc:287] T 32fad37a28e14903b8bde5aac6ee4e69 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.501s user 0.011s sys 0.000s
I20260810 23:32:46.934163 12240 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
I20260810 23:32:46.936106 12237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e
I20260810 23:32:46.937903 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440de7d9de8d4be9a5322e1fe3388b9b" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:46.939010 12805 raft_consensus.cc:3037] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:46.942647 12698 leader_election.cc:341] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:46.943137 12698 leader_election.cc:302] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:46.945883 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e with delete_type TABLET_DATA_DELETED (replaced by tablet d97b1c97910e450b8bc42e0c7a970ce0) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:46.946766 12237 catalog_manager.cc:6565] ReplaceTablet: tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e deleted and replaced by tablet d97b1c97910e450b8bc42e0c7a970ce0
I20260810 23:32:46.947610 12237 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 245) took 1467 ms. Trace:
I20260810 23:32:46.956187 14278 raft_consensus.cc:2726] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:46.964186 12234 catalog_manager.cc:5495] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129). New cstate: current_term: 1 leader_uuid: "191d120f4c7f44fc8475305a6d2e75f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:46.966626 12234 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet bd77526f1db4455291fde8a001092a09 (table default.Table_ac23e0995fcd414ead1b88406cf7d79e [id=963512beef0d4063b8e6a5738abd1d92]). Expected version 2 got 0
I20260810 23:32:46.920885 14178 raft_consensus.cc:482] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:46.969705 12236 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c [id=49a3a44dd7f9473d9c025d8690a00c1f] alter complete (version 2)
W20260810 23:32:46.970786 12229 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fb2ee1c9b41478da001aa67d7a7ad3c consensus state has no leader
I20260810 23:32:46.971380 12229 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 248) took 1345 ms. Trace:
I20260810 23:32:46.971686 12229 rpcz_store.cc:276] 0810 23:32:45.625939 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.627534 (+ 1595us) service_pool.cc:227] Handling call
0810 23:32:45.629332 (+ 1798us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:45.631405 (+ 2073us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:45.635159 (+ 3754us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:45.635380 (+ 221us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:45.808232 (+172852us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:46.971295 (+1163063us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:45.737947 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:45.761284 (+ 23337us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:45.761318 (+ 34us) write_op.cc:435] Acquired schema lock
0810 23:32:45.761336 (+ 18us) tablet.cc:660] Decoding operations
0810 23:32:45.761711 (+ 375us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:45.761760 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:45.761793 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:45.762016 (+ 223us) tablet.cc:699] Row locks acquired
0810 23:32:45.762056 (+ 40us) write_op.cc:260] PREPARE: finished
0810 23:32:45.762281 (+ 225us) write_op.cc:270] Start()
0810 23:32:45.762407 (+ 126us) write_op.cc:276] Timestamp: P: 1786404765762238 usec, L: 0
0810 23:32:45.762433 (+ 26us) op_driver.cc:331] REPLICATION: starting
0810 23:32:45.783360 (+ 20927us) log.cc:842] Serialized 405 byte log entry
0810 23:32:45.786559 (+ 3199us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:45.804256 (+ 17697us) op_driver.cc:444] REPLICATION: finished
0810 23:32:45.805034 (+ 778us) write_op.cc:301] APPLY: starting
0810 23:32:45.805153 (+ 119us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:45.805338 (+ 185us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:45.805400 (+ 62us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:45.805669 (+ 269us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:45.805959 (+ 290us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:45.806047 (+ 88us) write_op.cc:312] APPLY: finished
0810 23:32:45.806343 (+ 296us) log.cc:842] Serialized 23 byte log entry
0810 23:32:45.806601 (+ 258us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:45.806696 (+ 95us) write_op.cc:454] Released schema lock
0810 23:32:45.806736 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:45.807047 (+ 311us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:45.807092 (+ 45us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":493,"apply.run_cpu_time_us":2305,"apply.run_wall_time_us":2304,"num_ops":1,"prepare.queue_time_us":85222,"prepare.run_cpu_time_us":4355,"prepare.run_wall_time_us":83872,"raft.queue_time_us":130,"raft.run_cpu_time_us":931,"raft.run_wall_time_us":32907,"replication_time_us":41671,"spinlock_wait_cycles":46064384}]]}
I20260810 23:32:46.947985 12237 rpcz_store.cc:276] 0810 23:32:45.479748 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.538460 (+ 58712us) service_pool.cc:227] Handling call
0810 23:32:45.595010 (+ 56550us) spinlock_profiling.cc:243] Waited 32.9 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e9b81 00007f0ce6e44d37 00007f0ce6e44cc2 00007f0ce6e44bf1 00007f0ce6e44b6c 00007f0ce6e43e22 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:46.947450 (+1352440us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:46.115101 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:46.206370 (+ 91269us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:46.206462 (+ 92us) write_op.cc:435] Acquired schema lock
0810 23:32:46.206510 (+ 48us) tablet.cc:660] Decoding operations
0810 23:32:46.207379 (+ 869us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:46.207580 (+ 201us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:46.207652 (+ 72us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:46.208210 (+ 558us) tablet.cc:699] Row locks acquired
0810 23:32:46.208285 (+ 75us) write_op.cc:260] PREPARE: finished
0810 23:32:46.208651 (+ 366us) write_op.cc:270] Start()
0810 23:32:46.208856 (+ 205us) write_op.cc:276] Timestamp: P: 1786404766208566 usec, L: 0
0810 23:32:46.208875 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 23:32:46.210178 (+ 1303us) log.cc:842] Serialized 665 byte log entry
0810 23:32:46.762825 (+552647us) op_driver.cc:444] REPLICATION: finished
0810 23:32:46.865064 (+102239us) write_op.cc:301] APPLY: starting
0810 23:32:46.865241 (+ 177us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:46.865459 (+ 218us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:46.865678 (+ 219us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:46.866505 (+ 827us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:46.866864 (+ 359us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:46.866938 (+ 74us) write_op.cc:312] APPLY: finished
0810 23:32:46.867369 (+ 431us) log.cc:842] Serialized 29 byte log entry
0810 23:32:46.872202 (+ 4833us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:46.872328 (+ 126us) write_op.cc:454] Released schema lock
0810 23:32:46.872400 (+ 72us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":75587328,"child_traces":[["op",{"apply.queue_time_us":101863,"num_ops":2,"prepare.queue_time_us":71316,"prepare.run_cpu_time_us":5658,"prepare.run_wall_time_us":341061,"raft.queue_time_us":115,"raft.run_cpu_time_us":808,"raft.run_wall_time_us":808,"replication_time_us":553771,"thread_start_us":69505,"threads_started":1}]]}
I20260810 23:32:46.989537 14178 raft_consensus.cc:504] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:46.994055 14501 consensus_queue.cc:1035] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20260810 23:32:46.987881 13691 ts_tablet_manager.cc:1428] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.102s user 0.012s sys 0.001s
I20260810 23:32:47.004516 13935 raft_consensus.cc:2258] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:47.005429 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 132786us
I20260810 23:32:47.013955 13213 leader_election.cc:302] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:47.024746 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4f55ef709d473f89a16d32160eb61b" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:47.026441 12698 leader_election.cc:341] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:47.031652 13665 raft_consensus.cc:348] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:47.032516 13665 raft_consensus.cc:374] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:47.032769 13665 raft_consensus.cc:729] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:47.033459 13665 consensus_queue.cc:260] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:47.035274 13935 ts_tablet_manager.cc:1905] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:47.035768 13665 ts_tablet_manager.cc:1428] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.268s user 0.005s sys 0.001s
I20260810 23:32:47.043905 13665 tablet_bootstrap.cc:492] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:47.051275 13514 tablet_bootstrap.cc:492] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:47.051880 13514 ts_tablet_manager.cc:1397] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.323s user 0.011s sys 0.008s
I20260810 23:32:47.055107 13514 raft_consensus.cc:348] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.056327 13514 raft_consensus.cc:374] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:47.056277 13277 tablet_service.cc:1514] Processing DeleteTablet for tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e with delete_type TABLET_DATA_DELETED (replaced by tablet d97b1c97910e450b8bc42e0c7a970ce0) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:47.056653 13514 raft_consensus.cc:729] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:47.057878 13514 consensus_queue.cc:260] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.063223 12805 raft_consensus.cc:2454] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:47.065711 14178 leader_election.cc:288] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:47.082098 12958 leader_election.cc:302] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:47.085026 13691 tablet_bootstrap.cc:492] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
W20260810 23:32:47.097862 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8fb2ee1c9b41478da001aa67d7a7ad3c consensus state has no leader
I20260810 23:32:47.100747 13514 ts_tablet_manager.cc:1428] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.048s user 0.007s sys 0.000s
I20260810 23:32:47.102036 13514 tablet_bootstrap.cc:492] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:46.989248 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e with delete_type TABLET_DATA_DELETED (replaced by tablet d97b1c97910e450b8bc42e0c7a970ce0) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:47.111316 12228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 4cdec13ee70d402fa3a08119e7bf1313
I20260810 23:32:47.113075 12228 catalog_manager.cc:6565] ReplaceTablet: tablet 4cdec13ee70d402fa3a08119e7bf1313 deleted and replaced by tablet d603569c54ef4ffba9941d0cc3e0f97f
I20260810 23:32:47.113525 12228 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 235) took 2387 ms. Trace:
I20260810 23:32:47.115839 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4f55ef709d473f89a16d32160eb61b" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:46.911986 14427 consensus_queue.cc:1035] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.173s
I20260810 23:32:47.120103 12763 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdec13ee70d402fa3a08119e7bf1313 with delete_type TABLET_DATA_DELETED (replaced by tablet d603569c54ef4ffba9941d0cc3e0f97f) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:47.113865 12228 rpcz_store.cc:276] 0810 23:32:44.726542 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:44.957061 (+230519us) service_pool.cc:227] Handling call
0810 23:32:47.113449 (+2156388us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:46.456363 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:46.456874 (+ 511us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:46.456904 (+ 30us) write_op.cc:435] Acquired schema lock
0810 23:32:46.456917 (+ 13us) tablet.cc:660] Decoding operations
0810 23:32:46.457665 (+ 748us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:46.457711 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:46.457767 (+ 56us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:46.458266 (+ 499us) tablet.cc:699] Row locks acquired
0810 23:32:46.458308 (+ 42us) write_op.cc:260] PREPARE: finished
0810 23:32:46.458504 (+ 196us) write_op.cc:270] Start()
0810 23:32:46.458664 (+ 160us) write_op.cc:276] Timestamp: P: 1786404766458420 usec, L: 0
0810 23:32:46.458708 (+ 44us) op_driver.cc:331] REPLICATION: starting
0810 23:32:46.459636 (+ 928us) log.cc:842] Serialized 631 byte log entry
0810 23:32:46.853378 (+393742us) op_driver.cc:444] REPLICATION: finished
0810 23:32:46.913804 (+ 60426us) write_op.cc:301] APPLY: starting
0810 23:32:46.913879 (+ 75us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:46.914406 (+ 527us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:46.914949 (+ 543us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:47.108160 (+193211us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:47.108759 (+ 599us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:47.108882 (+ 123us) write_op.cc:312] APPLY: finished
0810 23:32:47.109307 (+ 425us) log.cc:842] Serialized 29 byte log entry
0810 23:32:47.110245 (+ 938us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:47.110369 (+ 124us) write_op.cc:454] Released schema lock
0810 23:32:47.110415 (+ 46us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":59961,"num_ops":2,"prepare.queue_time_us":401685,"prepare.run_cpu_time_us":3968,"prepare.run_wall_time_us":319309,"raft.queue_time_us":204,"raft.run_cpu_time_us":538,"raft.run_wall_time_us":538,"replication_time_us":394544,"thread_start_us":70887,"threads_started":1}]]}
I20260810 23:32:47.141603 14512 raft_consensus.cc:2781] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:47.142504 14512 raft_consensus.cc:686] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:47.143863 14512 consensus_queue.cc:237] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.155159 13691 tablet_bootstrap.cc:654] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:47.181855 14121 ts_tablet_manager.cc:1918] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:47.184695 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdec13ee70d402fa3a08119e7bf1313 with delete_type TABLET_DATA_DELETED (replaced by tablet d603569c54ef4ffba9941d0cc3e0f97f) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:47.217583 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 135578us
I20260810 23:32:47.224869 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdec13ee70d402fa3a08119e7bf1313 with delete_type TABLET_DATA_DELETED (replaced by tablet d603569c54ef4ffba9941d0cc3e0f97f) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:47.219841 14216 raft_consensus.cc:482] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:47.296693 14216 raft_consensus.cc:504] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:47.300812 12700 leader_election.cc:341] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:47.301323 12700 leader_election.cc:302] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:47.303014 14417 raft_consensus.cc:2726] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:47.262451 12952 consensus_peers.cc:487] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:47.305972 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 43784us
I20260810 23:32:47.307402 14216 leader_election.cc:288] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:47.309289 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d697e7533af4fc0a15cb41f3fcce4db" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:47.310251 12805 raft_consensus.cc:2454] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:47.311160 13691 tablet_bootstrap.cc:492] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:47.311810 13691 ts_tablet_manager.cc:1397] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.228s user 0.010s sys 0.014s
I20260810 23:32:47.312412 12958 leader_election.cc:302] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:47.313863 14216 raft_consensus.cc:2781] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:47.314728 14216 raft_consensus.cc:482] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:47.315346 14216 raft_consensus.cc:3037] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:47.316902 13514 tablet_bootstrap.cc:654] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:47.318711 14121 log.cc:1196] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/18b29cda94584badb1225477ec2bd1bc
I20260810 23:32:47.320248 14121 ts_tablet_manager.cc:1939] T 18b29cda94584badb1225477ec2bd1bc P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:47.342809 13935 ts_tablet_manager.cc:1918] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:47.343317 13935 log.cc:1196] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/b261673faa964b65af3576c89e302864
I20260810 23:32:47.321620 13974 ts_tablet_manager.cc:1918] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:47.346428 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d697e7533af4fc0a15cb41f3fcce4db" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:47.346112 13974 log.cc:1196] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/a0ba75f2c2b54ec1968db8feeacf8383
I20260810 23:32:47.354650 13974 ts_tablet_manager.cc:1939] T a0ba75f2c2b54ec1968db8feeacf8383 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:47.369820 12226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet f47b8b57d5b44a018bc2d7b3f976d803
I20260810 23:32:47.374724 12763 tablet_service.cc:1514] Processing DeleteTablet for tablet f47b8b57d5b44a018bc2d7b3f976d803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:47.380636 12226 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 252) took 1551 ms. Trace:
W20260810 23:32:47.382916 12952 leader_election.cc:341] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:47.384475 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet f47b8b57d5b44a018bc2d7b3f976d803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:47.381233 12226 rpcz_store.cc:276] 0810 23:32:45.828778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.913725 (+ 84947us) service_pool.cc:227] Handling call
0810 23:32:45.916963 (+ 3238us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7466 00007f0ce6e28b34 00007f0ce6e28ac2 00007f0ce6e289f1 00007f0ce6e2896c 00007f0ce6e27c22 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:45.918113 (+ 1150us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:45.918599 (+ 486us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:45.918739 (+ 140us) catalog_manager.cc:2761] Locking tablets
0810 23:32:46.037709 (+118970us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:47.369326 (+1331617us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:47.369593 (+ 267us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:47.369718 (+ 125us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:47.380507 (+ 10789us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:46.775913 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:46.869374 (+ 93461us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:46.869410 (+ 36us) write_op.cc:435] Acquired schema lock
0810 23:32:46.869474 (+ 64us) tablet.cc:660] Decoding operations
0810 23:32:46.870139 (+ 665us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:46.870237 (+ 98us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:46.870284 (+ 47us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:46.870961 (+ 677us) tablet.cc:699] Row locks acquired
0810 23:32:46.871005 (+ 44us) write_op.cc:260] PREPARE: finished
0810 23:32:46.871310 (+ 305us) write_op.cc:270] Start()
0810 23:32:46.871519 (+ 209us) write_op.cc:276] Timestamp: P: 1786404766871223 usec, L: 0
0810 23:32:46.871567 (+ 48us) op_driver.cc:331] REPLICATION: starting
0810 23:32:46.993178 (+121611us) log.cc:842] Serialized 734 byte log entry
0810 23:32:46.997729 (+ 4551us) spinlock_profiling.cc:243] Waited 54 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:47.363954 (+366225us) op_driver.cc:444] REPLICATION: finished
0810 23:32:47.365614 (+ 1660us) write_op.cc:301] APPLY: starting
0810 23:32:47.365762 (+ 148us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:47.365976 (+ 214us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:47.366015 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:47.366551 (+ 536us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:47.366835 (+ 284us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:47.366998 (+ 163us) write_op.cc:312] APPLY: finished
0810 23:32:47.367436 (+ 438us) log.cc:842] Serialized 29 byte log entry
0810 23:32:47.367708 (+ 272us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:47.367850 (+ 142us) write_op.cc:454] Released schema lock
0810 23:32:47.367911 (+ 61us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:47.368843 (+ 932us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:47.368909 (+ 66us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":5417856,"child_traces":[["op",{"apply.queue_time_us":1293,"apply.run_cpu_time_us":2935,"apply.run_wall_time_us":3547,"num_ops":2,"prepare.queue_time_us":735735,"prepare.run_cpu_time_us":8220,"prepare.run_wall_time_us":359307,"raft.queue_time_us":135178,"raft.run_cpu_time_us":909,"raft.run_wall_time_us":909,"replication_time_us":492192,"spinlock_wait_cycles":124157184}]]}
I20260810 23:32:47.390326 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_2e4541bbe3aa45c1a1c85533c2ed79f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:47.392594 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e4541bbe3aa45c1a1c85533c2ed79f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:47.345593 13665 tablet_bootstrap.cc:654] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:47.368842 14476 raft_consensus.cc:2726] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:47.378600 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet f47b8b57d5b44a018bc2d7b3f976d803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:47.411293 13935 ts_tablet_manager.cc:1939] T b261673faa964b65af3576c89e302864 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:47.415629 13691 raft_consensus.cc:348] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.417075 13691 raft_consensus.cc:374] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:47.421875 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 152) took 4261 ms. Trace:
I20260810 23:32:47.422281 13935 rpcz_store.cc:276] 0810 23:32:43.156614 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.156835 (+ 221us) service_pool.cc:227] Handling call
0810 23:32:46.717142 (+3560307us) ts_tablet_manager.cc:1145] Deleting tablet b261673faa964b65af3576c89e302864
0810 23:32:46.717207 (+ 65us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:46.897745 (+180538us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54000f3054. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3c5352 00007f3cde7db26c 00007f3cdf3e8cfa 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49 00007f3cdf3ffd11 00007f3cdf3fef3d 0000000000580784
0810 23:32:47.003857 (+106112us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b58001004b0. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde338f63 00007f3cde39caba 00007f3cde3c5375 00007f3cde7db26c 00007f3cdf3e8cfa 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49 00007f3cdf3ffd11
0810 23:32:47.083813 (+ 79956us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:47.342499 (+258686us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:47.418292 (+ 75793us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":643108992,"tablet-delete.queue_time_us":3528588}
I20260810 23:32:47.437870 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet b261673faa964b65af3576c89e302864 (table default.Table_adf76678e17a494185985a544def0f2d [id=5a377948adc7475c8c98141deaf1c274]) successfully deleted
W20260810 23:32:47.440476 12215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111) [suppressed 5 similar messages]
I20260810 23:32:47.456892 13935 tablet_replica.cc:330] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:47.445264 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 160) took 3275 ms. Trace:
I20260810 23:32:47.459574 13974 rpcz_store.cc:276] 0810 23:32:44.170897 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:44.171088 (+ 191us) service_pool.cc:227] Handling call
0810 23:32:46.598113 (+2427025us) ts_tablet_manager.cc:1145] Deleting tablet a0ba75f2c2b54ec1968db8feeacf8383
0810 23:32:46.648640 (+ 50527us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:47.094965 (+446325us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:47.321243 (+226278us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:47.445128 (+123885us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2421773}
I20260810 23:32:47.461335 13974 tablet_replica.cc:330] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:47.462384 13974 raft_consensus.cc:2229] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:47.463034 13974 raft_consensus.cc:2258] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:47.471261 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet a0ba75f2c2b54ec1968db8feeacf8383 (table default.Table_dd2cd30e5e2349de9180be01866dc513 [id=8754da7be72b4eef80daacdcdfde82a7]) successfully deleted
I20260810 23:32:47.480654 13935 raft_consensus.cc:2229] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:47.417567 14427 consensus_queue.cc:1035] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:47.427367 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 136) took 5829 ms. Trace:
I20260810 23:32:47.487462 14121 rpcz_store.cc:276] 0810 23:32:41.597753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.600058 (+ 2305us) service_pool.cc:227] Handling call
0810 23:32:46.310708 (+4710650us) ts_tablet_manager.cc:1145] Deleting tablet 18b29cda94584badb1225477ec2bd1bc
0810 23:32:46.310772 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:47.021659 (+710887us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:47.181497 (+159838us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:47.427252 (+245755us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30388,"tablet-delete.queue_time_us":4709539}
I20260810 23:32:47.489044 13514 tablet_bootstrap.cc:492] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:47.489576 13514 ts_tablet_manager.cc:1397] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.388s user 0.012s sys 0.009s
I20260810 23:32:47.489650 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 18b29cda94584badb1225477ec2bd1bc (table default.Table_b6ab31c2c3cb48fe84b341db2abba9ff [id=74d3cc5da04c4e098ac075abf30a8a72]) successfully deleted
I20260810 23:32:47.492697 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 137) took 5850 ms. Trace:
I20260810 23:32:47.493440 14121 rpcz_store.cc:276] 0810 23:32:41.641906 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.642217 (+ 311us) service_pool.cc:227] Handling call
0810 23:32:47.492152 (+5849935us) ts_tablet_manager.cc:1145] Deleting tablet 423e8d583d704fa9834026ebcd57e17e
0810 23:32:47.492228 (+ 76us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:47.492613 (+ 385us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5848471}
I20260810 23:32:47.461236 14516 raft_consensus.cc:482] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:32:47.497622 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 423e8d583d704fa9834026ebcd57e17e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 423e8d583d704fa9834026ebcd57e17e already in progress: creating tablet
I20260810 23:32:47.499505 14121 tablet_replica.cc:330] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:47.536545 14501 raft_consensus.cc:482] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:47.537217 14501 raft_consensus.cc:504] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.542035 14501 leader_election.cc:288] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:47.543260 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db363eb0f98d460eb6f5a577233d8ac0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:47.543605 14121 raft_consensus.cc:2229] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:47.494959 14516 raft_consensus.cc:504] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.545404 13935 raft_consensus.cc:2258] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:47.546283 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db363eb0f98d460eb6f5a577233d8ac0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:47.546993 12698 leader_election.cc:341] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:47.547447 13059 raft_consensus.cc:2454] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:32:47.549347 12700 leader_election.cc:302] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:47.550364 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42517f11422b42c39db282e2c5f584a3" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:47.550997 14501 raft_consensus.cc:2781] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:47.500586 13691 raft_consensus.cc:729] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:47.551501 14501 raft_consensus.cc:482] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:47.553347 14501 raft_consensus.cc:3037] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:47.556826 14121 raft_consensus.cc:2258] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:47.557610 13691 consensus_queue.cc:260] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:47.558528 12958 leader_election.cc:341] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 42517f11422b42c39db282e2c5f584a3
I20260810 23:32:47.559104 13974 ts_tablet_manager.cc:1905] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:47.561590 12952 leader_election.cc:341] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:47.562417 12952 leader_election.cc:302] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:47.563838 14319 raft_consensus.cc:2726] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:47.559067 14516 leader_election.cc:288] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:47.567984 13935 ts_tablet_manager.cc:1905] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:47.573943 13691 ts_tablet_manager.cc:1428] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.262s user 0.006s sys 0.001s
I20260810 23:32:47.541596 14216 raft_consensus.cc:504] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:47.594763 12235 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 253) took 1630 ms. Trace:
I20260810 23:32:47.595197 12235 rpcz_store.cc:276] 0810 23:32:45.964655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:46.051703 (+ 87048us) service_pool.cc:227] Handling call
0810 23:32:46.055589 (+ 3886us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:46.058931 (+ 3342us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:47.593965 (+1535034us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:47.594350 (+ 385us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:47.594678 (+ 328us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:47.135378 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:47.135876 (+ 498us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:47.135908 (+ 32us) write_op.cc:435] Acquired schema lock
0810 23:32:47.135922 (+ 14us) tablet.cc:660] Decoding operations
0810 23:32:47.136521 (+ 599us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:47.136606 (+ 85us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:47.136640 (+ 34us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:47.137042 (+ 402us) tablet.cc:699] Row locks acquired
0810 23:32:47.137065 (+ 23us) write_op.cc:260] PREPARE: finished
0810 23:32:47.137419 (+ 354us) write_op.cc:270] Start()
0810 23:32:47.137548 (+ 129us) write_op.cc:276] Timestamp: P: 1786404767137312 usec, L: 0
0810 23:32:47.137566 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 23:32:47.138837 (+ 1271us) log.cc:842] Serialized 433 byte log entry
0810 23:32:47.585432 (+446595us) op_driver.cc:444] REPLICATION: finished
0810 23:32:47.586668 (+ 1236us) write_op.cc:301] APPLY: starting
0810 23:32:47.586788 (+ 120us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:47.587028 (+ 240us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:47.587205 (+ 177us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:47.587959 (+ 754us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:47.588221 (+ 262us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:47.588296 (+ 75us) write_op.cc:312] APPLY: finished
0810 23:32:47.588675 (+ 379us) log.cc:842] Serialized 29 byte log entry
0810 23:32:47.588978 (+ 303us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:47.589081 (+ 103us) write_op.cc:454] Released schema lock
0810 23:32:47.589145 (+ 64us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:47.589512 (+ 367us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:47.589589 (+ 77us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":915,"apply.run_cpu_time_us":3162,"apply.run_wall_time_us":3161,"num_ops":2,"prepare.queue_time_us":1046640,"prepare.run_cpu_time_us":3809,"prepare.run_wall_time_us":37077,"replication_time_us":447716}]]}
I20260810 23:32:47.600510 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d697e7533af4fc0a15cb41f3fcce4db" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:47.601264 12802 raft_consensus.cc:3037] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:47.571873 14121 ts_tablet_manager.cc:1905] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:47.607964 12224 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc7adcb46ae0472384b3bb883ea58a74 has no consensus state
I20260810 23:32:47.608928 12224 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 254) took 1640 ms. Trace:
I20260810 23:32:47.609325 12224 rpcz_store.cc:276] 0810 23:32:45.969021 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:46.123381 (+154360us) service_pool.cc:227] Handling call
0810 23:32:46.132881 (+ 9500us) spinlock_profiling.cc:243] Waited 8.34 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e77d6 00007f0ce6e2d634 00007f0ce6e2d5c2 00007f0ce6e2d4f1 00007f0ce6e2d46c 00007f0ce6e2c722 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:46.135252 (+ 2371us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:46.136082 (+ 830us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:46.136763 (+ 681us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:46.136995 (+ 232us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:47.606949 (+1469954us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:47.608837 (+ 1888us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:47.172866 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:47.173936 (+ 1070us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:47.173999 (+ 63us) write_op.cc:435] Acquired schema lock
0810 23:32:47.174044 (+ 45us) tablet.cc:660] Decoding operations
0810 23:32:47.174782 (+ 738us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:47.174870 (+ 88us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:47.174899 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:47.175255 (+ 356us) tablet.cc:699] Row locks acquired
0810 23:32:47.175291 (+ 36us) write_op.cc:260] PREPARE: finished
0810 23:32:47.175613 (+ 322us) write_op.cc:270] Start()
0810 23:32:47.175836 (+ 223us) write_op.cc:276] Timestamp: P: 1786404767175483 usec, L: 0
0810 23:32:47.175898 (+ 62us) op_driver.cc:331] REPLICATION: starting
0810 23:32:47.233422 (+ 57524us) log.cc:842] Serialized 405 byte log entry
0810 23:32:47.585776 (+352354us) op_driver.cc:444] REPLICATION: finished
0810 23:32:47.590005 (+ 4229us) write_op.cc:301] APPLY: starting
0810 23:32:47.590091 (+ 86us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:47.590279 (+ 188us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:47.590422 (+ 143us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:47.590662 (+ 240us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:47.590951 (+ 289us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:47.591051 (+ 100us) write_op.cc:312] APPLY: finished
0810 23:32:47.591365 (+ 314us) log.cc:842] Serialized 23 byte log entry
0810 23:32:47.591594 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:47.591677 (+ 83us) write_op.cc:454] Released schema lock
0810 23:32:47.591731 (+ 54us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":19189632,"child_traces":[["op",{"apply.queue_time_us":4043,"num_ops":1,"prepare.queue_time_us":1034183,"prepare.run_cpu_time_us":4815,"prepare.run_wall_time_us":103427,"replication_time_us":409767,"threads_started":1}]]}
I20260810 23:32:47.612926 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d697e7533af4fc0a15cb41f3fcce4db" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
W20260810 23:32:47.619753 12952 leader_election.cc:341] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:47.621457 12952 consensus_peers.cc:487] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:47.622720 13691 tablet_bootstrap.cc:492] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:47.627604 14216 leader_election.cc:288] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:47.643131 13665 tablet_bootstrap.cc:492] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:47.644004 13665 ts_tablet_manager.cc:1397] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.600s user 0.012s sys 0.011s
I20260810 23:32:47.654235 14476 raft_consensus.cc:482] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:47.654772 14476 raft_consensus.cc:504] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:47.656929 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_7751e2b5b7db4c62beba2c5564a6affa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:47.659260 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7751e2b5b7db4c62beba2c5564a6affa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:47.677124 13691 tablet_bootstrap.cc:654] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:47.659375 13665 raft_consensus.cc:348] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:47.696892 14476 leader_election.cc:288] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
W20260810 23:32:47.699822 13212 leader_election.cc:341] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Not found: Tablet not found: 124259164adf404c9cbe5f4e0d5e16c0
I20260810 23:32:47.695026 13665 raft_consensus.cc:374] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 23:32:47.711812 13213 leader_election.cc:341] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 124259164adf404c9cbe5f4e0d5e16c0
I20260810 23:32:47.705411 13665 raft_consensus.cc:729] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:47.727706 14121 ts_tablet_manager.cc:1918] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:47.730363 13665 consensus_queue.cc:260] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:47.728345 14121 log.cc:1196] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/3d596299c82b4ff1ac151d68b97ff55b
I20260810 23:32:47.735136 14121 ts_tablet_manager.cc:1939] T 3d596299c82b4ff1ac151d68b97ff55b P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:47.712344 13213 leader_election.cc:302] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:47.738067 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 139) took 6091 ms. Trace:
I20260810 23:32:47.739262 14476 raft_consensus.cc:2726] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:47.744076 13665 ts_tablet_manager.cc:1428] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.091s user 0.006s sys 0.001s
I20260810 23:32:47.736605 13514 raft_consensus.cc:348] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.745151 13514 raft_consensus.cc:374] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:47.745501 13514 raft_consensus.cc:729] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:47.753140 13665 tablet_bootstrap.cc:492] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:47.754402 13691 tablet_bootstrap.cc:492] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:47.755025 13691 ts_tablet_manager.cc:1397] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.178s user 0.016s sys 0.010s
I20260810 23:32:47.751361 13514 consensus_queue.cc:260] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.738495 14121 rpcz_store.cc:276] 0810 23:32:41.647227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.647412 (+ 185us) service_pool.cc:227] Handling call
0810 23:32:47.499257 (+5851845us) ts_tablet_manager.cc:1145] Deleting tablet 3d596299c82b4ff1ac151d68b97ff55b
0810 23:32:47.499327 (+ 70us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:47.687055 (+187728us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:47.715934 (+ 28879us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:47.737986 (+ 22052us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22610,"tablet-delete.queue_time_us":5850695}
I20260810 23:32:47.766637 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 3d596299c82b4ff1ac151d68b97ff55b (table default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]) successfully deleted
I20260810 23:32:47.769176 14121 tablet_replica.cc:330] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:47.770149 14121 raft_consensus.cc:2229] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:47.770282 13514 ts_tablet_manager.cc:1428] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.280s user 0.010s sys 0.012s
I20260810 23:32:47.770823 14121 raft_consensus.cc:2258] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:47.771549 13514 tablet_bootstrap.cc:492] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:47.781766 13691 raft_consensus.cc:348] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.782910 13691 raft_consensus.cc:374] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:47.783227 13691 raft_consensus.cc:729] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:47.785876 13665 tablet_bootstrap.cc:654] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:47.787216 14121 ts_tablet_manager.cc:1905] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:47.803570 13514 tablet_bootstrap.cc:654] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:47.808398 13691 consensus_queue.cc:260] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.818089 13665 tablet_bootstrap.cc:492] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:47.818624 13665 ts_tablet_manager.cc:1397] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.066s user 0.013s sys 0.004s
I20260810 23:32:47.828648 13691 ts_tablet_manager.cc:1428] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.073s user 0.012s sys 0.001s
I20260810 23:32:47.821646 13665 raft_consensus.cc:348] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.834914 13665 raft_consensus.cc:374] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:47.835357 13665 raft_consensus.cc:729] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:47.836558 13665 consensus_queue.cc:260] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.852914 13691 tablet_bootstrap.cc:492] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:47.853847 12223 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 255) took 1784 ms. Trace:
I20260810 23:32:47.854269 12223 rpcz_store.cc:276] 0810 23:32:46.067562 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:46.141034 (+ 73472us) service_pool.cc:227] Handling call
0810 23:32:46.145732 (+ 4698us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:46.193657 (+ 47925us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:47.842622 (+1648965us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:47.851271 (+ 8649us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:47.851725 (+ 454us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:47.276903 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:47.278477 (+ 1574us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:47.278512 (+ 35us) write_op.cc:435] Acquired schema lock
0810 23:32:47.278526 (+ 14us) tablet.cc:660] Decoding operations
0810 23:32:47.279284 (+ 758us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:47.279390 (+ 106us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:47.279464 (+ 74us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:47.280011 (+ 547us) tablet.cc:699] Row locks acquired
0810 23:32:47.336874 (+ 56863us) write_op.cc:260] PREPARE: finished
0810 23:32:47.337173 (+ 299us) write_op.cc:270] Start()
0810 23:32:47.337383 (+ 210us) write_op.cc:276] Timestamp: P: 1786404767337100 usec, L: 0
0810 23:32:47.337401 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 23:32:47.338539 (+ 1138us) log.cc:842] Serialized 433 byte log entry
0810 23:32:47.648572 (+310033us) op_driver.cc:444] REPLICATION: finished
0810 23:32:47.691333 (+ 42761us) write_op.cc:301] APPLY: starting
0810 23:32:47.691440 (+ 107us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:47.691716 (+ 276us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:47.692052 (+ 336us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:47.693034 (+ 982us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:47.693300 (+ 266us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:47.693374 (+ 74us) write_op.cc:312] APPLY: finished
0810 23:32:47.693697 (+ 323us) log.cc:842] Serialized 29 byte log entry
0810 23:32:47.833901 (+140204us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:47.834022 (+ 121us) write_op.cc:454] Released schema lock
0810 23:32:47.834062 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:47.841954 (+ 7892us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:47.841985 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":42288,"apply.run_cpu_time_us":4527,"apply.run_wall_time_us":150960,"num_ops":2,"prepare.queue_time_us":1081284,"prepare.run_cpu_time_us":5700,"prepare.run_wall_time_us":127602,"replication_time_us":310885}]]}
I20260810 23:32:47.865928 13665 ts_tablet_manager.cc:1428] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.047s user 0.009s sys 0.001s
I20260810 23:32:47.881865 13691 tablet_bootstrap.cc:654] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:47.899817 12226 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 256) took 1778 ms. Trace:
I20260810 23:32:47.900312 12226 rpcz_store.cc:276] 0810 23:32:46.121363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:47.389580 (+1268217us) service_pool.cc:227] Handling call
0810 23:32:47.393247 (+ 3667us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:47.415451 (+ 22204us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:47.899049 (+483598us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:47.899413 (+ 364us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:47.899736 (+ 323us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:47.633451 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:47.634471 (+ 1020us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:47.634512 (+ 41us) write_op.cc:435] Acquired schema lock
0810 23:32:47.634580 (+ 68us) tablet.cc:660] Decoding operations
0810 23:32:47.635242 (+ 662us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:47.635305 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:47.635350 (+ 45us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:47.635900 (+ 550us) tablet.cc:699] Row locks acquired
0810 23:32:47.635954 (+ 54us) write_op.cc:260] PREPARE: finished
0810 23:32:47.793716 (+157762us) write_op.cc:270] Start()
0810 23:32:47.793941 (+ 225us) write_op.cc:276] Timestamp: P: 1786404767793618 usec, L: 0
0810 23:32:47.793964 (+ 23us) op_driver.cc:331] REPLICATION: starting
0810 23:32:47.834163 (+ 40199us) log.cc:842] Serialized 433 byte log entry
0810 23:32:47.868861 (+ 34698us) spinlock_profiling.cc:243] Waited 38.1 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:47.890733 (+ 21872us) op_driver.cc:444] REPLICATION: finished
0810 23:32:47.896069 (+ 5336us) write_op.cc:301] APPLY: starting
0810 23:32:47.896141 (+ 72us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:47.896393 (+ 252us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:47.896414 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:47.897337 (+ 923us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:47.897540 (+ 203us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:47.897595 (+ 55us) write_op.cc:312] APPLY: finished
0810 23:32:47.897840 (+ 245us) log.cc:842] Serialized 29 byte log entry
0810 23:32:47.898665 (+ 825us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:47.898752 (+ 87us) write_op.cc:454] Released schema lock
0810 23:32:47.898795 (+ 43us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4760,"num_ops":2,"prepare.queue_time_us":178622,"prepare.run_cpu_time_us":6123,"prepare.run_wall_time_us":268588,"raft.queue_time_us":518,"raft.run_cpu_time_us":1119,"raft.run_wall_time_us":20087,"replication_time_us":96643,"spinlock_wait_cycles":87706496}]]}
I20260810 23:32:47.909735 13665 tablet_bootstrap.cc:492] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:47.935110 13514 tablet_bootstrap.cc:492] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:47.890448 14501 raft_consensus.cc:504] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.935945 13514 ts_tablet_manager.cc:1397] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.165s user 0.014s sys 0.013s
I20260810 23:32:47.940070 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db363eb0f98d460eb6f5a577233d8ac0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:47.941067 13062 raft_consensus.cc:3037] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:47.957762 13514 raft_consensus.cc:348] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:47.960635 13514 raft_consensus.cc:374] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:47.964172 13062 raft_consensus.cc:2454] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:47.966337 13665 tablet_bootstrap.cc:654] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:47.965787 12700 leader_election.cc:302] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:47.972154 14178 raft_consensus.cc:2781] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:47.972759 12802 raft_consensus.cc:2454] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:47.983603 13514 raft_consensus.cc:729] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:47.988091 14501 leader_election.cc:288] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:47.989706 13691 tablet_bootstrap.cc:492] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:47.990322 13691 ts_tablet_manager.cc:1397] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.144s user 0.012s sys 0.005s
I20260810 23:32:47.967154 14121 ts_tablet_manager.cc:1918] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:47.997054 14121 log.cc:1196] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/b8986e59b2314bd69b9b1de36f1f2ba7
I20260810 23:32:47.995088 13514 consensus_queue.cc:260] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.010646 13514 ts_tablet_manager.cc:1428] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.074s user 0.004s sys 0.004s
I20260810 23:32:48.012225 14178 raft_consensus.cc:686] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Becoming Leader. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Running, Role: LEADER
I20260810 23:32:48.013672 14178 consensus_queue.cc:237] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.017638 12958 leader_election.cc:302] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:48.018868 14512 raft_consensus.cc:2781] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:48.021607 14512 raft_consensus.cc:686] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:48.023674 14512 consensus_queue.cc:237] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:48.028728 12224 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 257) took 1696 ms. Trace:
I20260810 23:32:48.029848 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db363eb0f98d460eb6f5a577233d8ac0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:48.029238 12224 rpcz_store.cc:276] 0810 23:32:46.332606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:47.617670 (+1285064us) service_pool.cc:227] Handling call
0810 23:32:47.656639 (+ 38969us) spinlock_profiling.cc:243] Waited 37.1 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7156 00007f0ce6e24034 00007f0ce6e23fc2 00007f0ce6e23ef1 00007f0ce6e23e6c 00007f0ce6e23122 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:47.659714 (+ 3075us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:47.814002 (+154288us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b0c00031e88. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce0206f11 00007f0ce031175f 00007f0ce0311599 00007f0ce91c94df 00007f0ce91c7683 00007f0ce91c4ecc 00007f0ce92e718f 00007f0ce6e24034 00007f0ce6e23fc2 00007f0ce6e23ef1 00007f0ce6e23e6c
0810 23:32:47.814930 (+ 928us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:48.020402 (+205472us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:48.028304 (+ 7902us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:48.028644 (+ 340us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:47.933022 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:47.934226 (+ 1204us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:47.934299 (+ 73us) write_op.cc:435] Acquired schema lock
0810 23:32:47.934340 (+ 41us) tablet.cc:660] Decoding operations
0810 23:32:47.934961 (+ 621us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:47.935058 (+ 97us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:47.935115 (+ 57us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:47.935558 (+ 443us) tablet.cc:699] Row locks acquired
0810 23:32:47.935591 (+ 33us) write_op.cc:260] PREPARE: finished
0810 23:32:47.935871 (+ 280us) write_op.cc:270] Start()
0810 23:32:47.960839 (+ 24968us) write_op.cc:276] Timestamp: P: 1786404767935780 usec, L: 0
0810 23:32:47.960870 (+ 31us) op_driver.cc:331] REPLICATION: starting
0810 23:32:47.967536 (+ 6666us) log.cc:842] Serialized 433 byte log entry
0810 23:32:47.987203 (+ 19667us) spinlock_profiling.cc:243] Waited 5.69 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:48.015390 (+ 28187us) op_driver.cc:444] REPLICATION: finished
0810 23:32:48.016146 (+ 756us) write_op.cc:301] APPLY: starting
0810 23:32:48.016236 (+ 90us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:48.016434 (+ 198us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:48.016461 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:48.017131 (+ 670us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:48.017406 (+ 275us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:48.017613 (+ 207us) write_op.cc:312] APPLY: finished
0810 23:32:48.017992 (+ 379us) log.cc:842] Serialized 29 byte log entry
0810 23:32:48.018307 (+ 315us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:48.018461 (+ 154us) write_op.cc:454] Released schema lock
0810 23:32:48.018497 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:48.018846 (+ 349us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:48.018953 (+ 107us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":377198592,"child_traces":[["op",{"apply.queue_time_us":345,"apply.run_cpu_time_us":3106,"apply.run_wall_time_us":3190,"num_ops":2,"prepare.queue_time_us":165,"prepare.run_cpu_time_us":5476,"prepare.run_wall_time_us":73258,"raft.queue_time_us":35171,"raft.run_cpu_time_us":853,"raft.run_wall_time_us":1864,"replication_time_us":54362,"spinlock_wait_cycles":13095296}]]}
W20260810 23:32:48.036837 12698 leader_election.cc:341] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:47.998957 14121 ts_tablet_manager.cc:1939] T b8986e59b2314bd69b9b1de36f1f2ba7 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:48.039369 12224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_96019e28037b4d38aa9fc933a29c8884" } modify_external_catalogs: true
I20260810 23:32:48.040704 12224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_96019e28037b4d38aa9fc933a29c8884" } modify_external_catalogs: true
I20260810 23:32:48.060946 13691 raft_consensus.cc:348] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.061733 13691 raft_consensus.cc:374] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:48.061987 13691 raft_consensus.cc:729] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:48.064134 13691 consensus_queue.cc:260] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.068082 13514 tablet_bootstrap.cc:492] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:32:48.085019 14565 log.cc:924] Time spent T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3: Append to log took a long time: real 0.054s user 0.003s sys 0.000s
I20260810 23:32:48.093469 13935 ts_tablet_manager.cc:1918] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:48.123588 13665 tablet_bootstrap.cc:492] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:48.124330 13665 ts_tablet_manager.cc:1397] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.258s user 0.012s sys 0.005s
I20260810 23:32:48.128374 13935 log.cc:1196] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/440de7d9de8d4be9a5322e1fe3388b9b
I20260810 23:32:48.129688 13935 ts_tablet_manager.cc:1939] T 440de7d9de8d4be9a5322e1fe3388b9b P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
W20260810 23:32:48.138286 14574 log.cc:924] Time spent T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260810 23:32:48.139567 13691 ts_tablet_manager.cc:1428] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.149s user 0.009s sys 0.000s
I20260810 23:32:48.141186 13691 tablet_bootstrap.cc:492] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:48.143921 14427 raft_consensus.cc:482] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:48.146379 14427 raft_consensus.cc:504] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.149235 14512 raft_consensus.cc:482] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:48.149712 14427 leader_election.cc:288] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:48.151088 13974 ts_tablet_manager.cc:1918] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:48.151615 13974 log.cc:1196] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/18b29cda94584badb1225477ec2bd1bc
I20260810 23:32:48.152787 14571 raft_consensus.cc:482] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:48.153469 14571 raft_consensus.cc:504] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:48.156366 12958 leader_election.cc:341] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 133654008f5247b2ac87dcd09195f197
I20260810 23:32:48.160295 13665 raft_consensus.cc:348] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.161192 13665 raft_consensus.cc:374] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:48.161526 13665 raft_consensus.cc:729] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:48.161906 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30bc9bd4e1f462b8d3be7f8a2f28f1e" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:48.162336 13665 consensus_queue.cc:260] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.164973 13974 ts_tablet_manager.cc:1939] T 18b29cda94584badb1225477ec2bd1bc P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:48.166759 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30bc9bd4e1f462b8d3be7f8a2f28f1e" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:48.165174 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133654008f5247b2ac87dcd09195f197" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:48.167446 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 165) took 3930 ms. Trace:
I20260810 23:32:48.167788 13331 raft_consensus.cc:2454] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:48.170545 13514 tablet_bootstrap.cc:654] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:48.179914 12952 leader_election.cc:302] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:48.150956 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 138) took 6504 ms. Trace:
I20260810 23:32:48.187971 14581 tablet.cc:1722] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:48.194468 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 157) took 4814 ms. Trace:
I20260810 23:32:48.194937 13935 rpcz_store.cc:276] 0810 23:32:43.380547 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.380713 (+ 166us) service_pool.cc:227] Handling call
0810 23:32:47.446874 (+4066161us) ts_tablet_manager.cc:1145] Deleting tablet 440de7d9de8d4be9a5322e1fe3388b9b
0810 23:32:47.456689 (+ 9815us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:47.911702 (+455013us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:48.093159 (+181457us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:48.194368 (+101209us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62991,"tablet-delete.queue_time_us":3758494}
I20260810 23:32:48.196977 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 440de7d9de8d4be9a5322e1fe3388b9b (table default.Table_3b251f8589be43f29704b8906d49f6cd [id=c56022cc8cad4c7cbefafad0782cefde]) successfully deleted
I20260810 23:32:48.198786 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 158) took 4817 ms. Trace:
I20260810 23:32:48.199272 13935 rpcz_store.cc:276] 0810 23:32:43.381181 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.381350 (+ 169us) service_pool.cc:227] Handling call
0810 23:32:48.198154 (+4816804us) ts_tablet_manager.cc:1145] Deleting tablet 4f3049bcdc6e43c795961c74f0d008f5
0810 23:32:48.198252 (+ 98us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:48.198716 (+ 464us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4509509}
I20260810 23:32:48.186735 14121 rpcz_store.cc:276] 0810 23:32:41.646743 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:41.649280 (+ 2537us) service_pool.cc:227] Handling call
0810 23:32:47.768927 (+6119647us) ts_tablet_manager.cc:1145] Deleting tablet b8986e59b2314bd69b9b1de36f1f2ba7
0810 23:32:47.768998 (+ 71us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:47.873882 (+104884us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:47.966858 (+ 92976us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:48.150877 (+184019us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38108,"tablet-delete.queue_time_us":6115444}
W20260810 23:32:48.201092 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 4f3049bcdc6e43c795961c74f0d008f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f3049bcdc6e43c795961c74f0d008f5 already in progress: creating tablet
I20260810 23:32:48.202730 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 156) took 4918 ms. Trace:
I20260810 23:32:48.204352 13935 rpcz_store.cc:276] 0810 23:32:43.284606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.284729 (+ 123us) service_pool.cc:227] Handling call
0810 23:32:48.202258 (+4917529us) ts_tablet_manager.cc:1145] Deleting tablet c45f9dfd03764d7fb1b6fb5e28b66bd3
0810 23:32:48.202336 (+ 78us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:48.202655 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4512814}
I20260810 23:32:48.205132 14121 tablet_replica.cc:330] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:48.206260 14121 raft_consensus.cc:2229] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:48.159546 14571 leader_election.cc:288] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:48.149803 14512 raft_consensus.cc:504] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.206990 13212 leader_election.cc:302] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:48.210018 14476 raft_consensus.cc:2726] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:48.211737 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b71d565669f40d8a6d06ab267069430" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:48.213979 14216 raft_consensus.cc:2781] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260810 23:32:48.214037 12952 leader_election.cc:341] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:48.217437 13665 ts_tablet_manager.cc:1428] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.093s user 0.007s sys 0.000s
I20260810 23:32:48.218837 13665 tablet_bootstrap.cc:492] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:48.167814 13974 rpcz_store.cc:276] 0810 23:32:44.236902 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:44.237049 (+ 147us) service_pool.cc:227] Handling call
0810 23:32:47.461101 (+3224052us) ts_tablet_manager.cc:1145] Deleting tablet 18b29cda94584badb1225477ec2bd1bc
0810 23:32:47.461173 (+ 72us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:47.927866 (+466693us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:48.150729 (+222863us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:48.167369 (+ 16640us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3219842}
I20260810 23:32:48.221978 13974 tablet_replica.cc:330] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:48.222910 13974 raft_consensus.cc:2229] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:48.223757 13974 raft_consensus.cc:2258] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:48.214620 14216 raft_consensus.cc:482] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:48.228976 14216 raft_consensus.cc:3037] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:48.229965 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8fb2ee1c9b41478da001aa67d7a7ad3c consensus state has no leader
W20260810 23:32:48.235632 12958 consensus_peers.cc:487] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 440de7d9de8d4be9a5322e1fe3388b9b. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:48.257113 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 because the tablet was not found. No further retry: Not found: Tablet not found: c45f9dfd03764d7fb1b6fb5e28b66bd3
W20260810 23:32:48.239998 14569 log.cc:924] Time spent T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3: Append to log took a long time: real 0.068s user 0.003s sys 0.003s
I20260810 23:32:48.257841 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b71d565669f40d8a6d06ab267069430" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:48.259022 13935 tablet_replica.cc:330] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:48.260165 13935 raft_consensus.cc:2229] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:48.260895 13935 raft_consensus.cc:2258] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:48.263489 13935 ts_tablet_manager.cc:1905] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:48.259112 12802 raft_consensus.cc:2454] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:48.270903 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 18b29cda94584badb1225477ec2bd1bc (table default.Table_b6ab31c2c3cb48fe84b341db2abba9ff [id=74d3cc5da04c4e098ac075abf30a8a72]) successfully deleted
I20260810 23:32:48.271584 13514 tablet_bootstrap.cc:492] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:48.271481 12958 leader_election.cc:302] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:48.272650 14512 leader_election.cc:288] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:48.287526 14121 raft_consensus.cc:2258] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:48.290550 13691 tablet_bootstrap.cc:654] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:48.268584 14571 raft_consensus.cc:482] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:48.291424 14571 raft_consensus.cc:504] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.293099 14121 ts_tablet_manager.cc:1905] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:48.293313 13974 ts_tablet_manager.cc:1905] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:48.273509 14589 tablet.cc:1722] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:48.298269 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ad5d63a2104995990a86a000963cdc" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:48.299167 13059 raft_consensus.cc:2454] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:48.300798 13212 leader_election.cc:302] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:48.302016 14582 raft_consensus.cc:2781] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:48.302455 14582 raft_consensus.cc:482] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:48.302866 14582 raft_consensus.cc:3037] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:48.302834 14216 raft_consensus.cc:504] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.304194 14427 raft_consensus.cc:2781] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:48.304594 14427 raft_consensus.cc:482] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:48.304936 14427 raft_consensus.cc:3037] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:48.306756 13665 tablet_bootstrap.cc:654] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:48.314554 14571 leader_election.cc:288] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:48.315239 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ad5d63a2104995990a86a000963cdc" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:48.315775 14216 leader_election.cc:288] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:48.316257 12802 raft_consensus.cc:2454] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
W20260810 23:32:48.317901 12958 leader_election.cc:341] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 133654008f5247b2ac87dcd09195f197
I20260810 23:32:48.318914 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133654008f5247b2ac87dcd09195f197" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:48.319041 14591 tablet.cc:1722] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:48.320093 13331 raft_consensus.cc:3037] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:48.339805 13514 ts_tablet_manager.cc:1397] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.286s user 0.010s sys 0.013s
I20260810 23:32:48.343600 13514 raft_consensus.cc:348] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:48.344396 13514 raft_consensus.cc:374] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:48.344671 13514 raft_consensus.cc:729] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:48.345451 13331 raft_consensus.cc:2454] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:48.353766 12952 leader_election.cc:302] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:48.355192 14216 raft_consensus.cc:2781] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:48.356072 14216 raft_consensus.cc:686] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:48.347683 14582 raft_consensus.cc:504] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.365244 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ad5d63a2104995990a86a000963cdc" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:48.365949 14582 leader_election.cc:288] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:48.365998 13062 raft_consensus.cc:3037] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:48.367189 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ad5d63a2104995990a86a000963cdc" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:48.367859 12805 raft_consensus.cc:3037] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:48.376490 13514 consensus_queue.cc:260] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:48.378345 14427 raft_consensus.cc:504] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.376492 12805 raft_consensus.cc:2454] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:48.383247 13213 leader_election.cc:302] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:48.384346 14582 raft_consensus.cc:2781] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:48.388985 14582 raft_consensus.cc:686] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:48.390202 14582 consensus_queue.cc:237] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.397305 14567 raft_consensus.cc:482] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:48.397825 14567 raft_consensus.cc:504] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:48.403644 14567 leader_election.cc:288] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:48.403944 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c29b11192604ac18c21b7bdc0238fb3" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:48.407143 12698 leader_election.cc:341] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:48.409165 13514 ts_tablet_manager.cc:1428] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.068s user 0.005s sys 0.001s
I20260810 23:32:48.365552 14216 consensus_queue.cc:237] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.412628 12228 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet bd77526f1db4455291fde8a001092a09 (table default.Table_ac23e0995fcd414ead1b88406cf7d79e [id=963512beef0d4063b8e6a5738abd1d92]). Expected version 2 got 0
I20260810 23:32:48.415383 14427 leader_election.cc:288] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
W20260810 23:32:48.388491 12952 consensus_peers.cc:487] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:48.414206 12228 catalog_manager.cc:5495] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129). New cstate: current_term: 1 leader_uuid: "191d120f4c7f44fc8475305a6d2e75f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:48.417301 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b71d565669f40d8a6d06ab267069430" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:48.417646 12228 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
I20260810 23:32:48.418028 12805 raft_consensus.cc:3037] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:48.401446 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c29b11192604ac18c21b7bdc0238fb3" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:48.422863 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b71d565669f40d8a6d06ab267069430" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:48.423424 13331 raft_consensus.cc:3037] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:48.424381 12700 leader_election.cc:341] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:48.424844 12700 leader_election.cc:302] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:48.426077 14567 raft_consensus.cc:2726] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:48.428843 12952 leader_election.cc:341] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:48.437772 13279 tablet_service.cc:1467] Processing CreateTablet for tablet 7276caee8a674131860ef17898c65308 (DEFAULT_TABLE table=default.Table_3afb998b53b2444d831fcfc512a9c665 [id=a02006db831d417597d63c0e7c9cad8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.439221 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7276caee8a674131860ef17898c65308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.412570 13691 tablet_bootstrap.cc:492] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:48.444576 13691 ts_tablet_manager.cc:1397] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.304s user 0.017s sys 0.008s
W20260810 23:32:48.447063 12958 consensus_peers.cc:487] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:48.447587 12218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.188:45477, user_credentials={real_user=slave}} blocked reactor thread for 94584.3us
I20260810 23:32:48.450198 13691 raft_consensus.cc:348] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.451097 13691 raft_consensus.cc:374] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:48.451370 13691 raft_consensus.cc:729] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:48.455690 12763 tablet_service.cc:1467] Processing CreateTablet for tablet 7276caee8a674131860ef17898c65308 (DEFAULT_TABLE table=default.Table_3afb998b53b2444d831fcfc512a9c665 [id=a02006db831d417597d63c0e7c9cad8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.456460 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 7276caee8a674131860ef17898c65308 (DEFAULT_TABLE table=default.Table_3afb998b53b2444d831fcfc512a9c665 [id=a02006db831d417597d63c0e7c9cad8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.457792 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7276caee8a674131860ef17898c65308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.457821 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7276caee8a674131860ef17898c65308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.460161 12805 raft_consensus.cc:2454] T 4b71d565669f40d8a6d06ab267069430 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:48.461632 12958 leader_election.cc:302] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:48.463161 14427 raft_consensus.cc:2781] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:48.463932 14427 raft_consensus.cc:686] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:48.464838 14427 consensus_queue.cc:237] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.475103 13514 tablet_bootstrap.cc:492] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:48.476490 13691 consensus_queue.cc:260] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.375953 13062 raft_consensus.cc:2454] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:48.482929 13691 ts_tablet_manager.cc:1428] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.038s user 0.006s sys 0.001s
I20260810 23:32:48.489466 13691 tablet_bootstrap.cc:492] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:48.518013 12764 tablet_service.cc:1467] Processing CreateTablet for tablet 95622c5782e443b9903c85f64852eaa9 (DEFAULT_TABLE table=default.Table_2ad7503d6e154e87a585df0b30bcbd1d [id=f8f38ab178fe42baab31717646974470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.518537 13007 tablet_service.cc:1467] Processing CreateTablet for tablet 95622c5782e443b9903c85f64852eaa9 (DEFAULT_TABLE table=default.Table_2ad7503d6e154e87a585df0b30bcbd1d [id=f8f38ab178fe42baab31717646974470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.519218 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95622c5782e443b9903c85f64852eaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.524665 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 95622c5782e443b9903c85f64852eaa9 (DEFAULT_TABLE table=default.Table_2ad7503d6e154e87a585df0b30bcbd1d [id=f8f38ab178fe42baab31717646974470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.527031 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95622c5782e443b9903c85f64852eaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.527925 13665 tablet_bootstrap.cc:492] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:48.528512 13665 ts_tablet_manager.cc:1397] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.310s user 0.012s sys 0.001s
I20260810 23:32:48.531451 13277 tablet_service.cc:1467] Processing CreateTablet for tablet 8163921fdc2c442cb335041c79de6d6d (DEFAULT_TABLE table=default.Table_bec275e3dbf54eb08d1df8d68e06b5a5 [id=6541682833df436f8f687c3a20fee458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.532738 12761 tablet_service.cc:1467] Processing CreateTablet for tablet 8163921fdc2c442cb335041c79de6d6d (DEFAULT_TABLE table=default.Table_bec275e3dbf54eb08d1df8d68e06b5a5 [id=6541682833df436f8f687c3a20fee458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.533511 13277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8163921fdc2c442cb335041c79de6d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.533546 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 8163921fdc2c442cb335041c79de6d6d (DEFAULT_TABLE table=default.Table_bec275e3dbf54eb08d1df8d68e06b5a5 [id=6541682833df436f8f687c3a20fee458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.535029 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8163921fdc2c442cb335041c79de6d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.537783 13007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95622c5782e443b9903c85f64852eaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.541266 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8163921fdc2c442cb335041c79de6d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.549970 13514 tablet_bootstrap.cc:654] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:48.560909 14601 log.cc:924] Time spent T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20260810 23:32:48.564450 13665 raft_consensus.cc:348] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.568172 13665 raft_consensus.cc:374] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:48.568598 13665 raft_consensus.cc:729] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:48.569602 13665 consensus_queue.cc:260] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.570094 13286 tablet_service.cc:1467] Processing CreateTablet for tablet ad8be93f60644e16b28453b2fd6c0285 (DEFAULT_TABLE table=default.Table_0f0715c81643462985ef73882b0ccb39 [id=5b00eb54477e4461a5710c99f2517d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.571230 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad8be93f60644e16b28453b2fd6c0285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.572366 13665 ts_tablet_manager.cc:1428] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.044s user 0.009s sys 0.000s
I20260810 23:32:48.573427 12756 tablet_service.cc:1467] Processing CreateTablet for tablet ad8be93f60644e16b28453b2fd6c0285 (DEFAULT_TABLE table=default.Table_0f0715c81643462985ef73882b0ccb39 [id=5b00eb54477e4461a5710c99f2517d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.574103 13008 tablet_service.cc:1467] Processing CreateTablet for tablet ad8be93f60644e16b28453b2fd6c0285 (DEFAULT_TABLE table=default.Table_0f0715c81643462985ef73882b0ccb39 [id=5b00eb54477e4461a5710c99f2517d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.575335 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad8be93f60644e16b28453b2fd6c0285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.576282 13008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad8be93f60644e16b28453b2fd6c0285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.587030 13691 tablet_bootstrap.cc:654] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:48.597295 13665 tablet_bootstrap.cc:492] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:48.605901 12768 tablet_service.cc:1467] Processing CreateTablet for tablet d5573b3e7e5b47f9885fc1807202d2e6 (DEFAULT_TABLE table=default.Table_1072812b5a8a49a1a6a1a94eb4e2b319 [id=b3a8515067ee4ce08f1d5dd8e72f56f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.607240 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5573b3e7e5b47f9885fc1807202d2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.608697 13023 tablet_service.cc:1467] Processing CreateTablet for tablet d5573b3e7e5b47f9885fc1807202d2e6 (DEFAULT_TABLE table=default.Table_1072812b5a8a49a1a6a1a94eb4e2b319 [id=b3a8515067ee4ce08f1d5dd8e72f56f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.609067 14567 raft_consensus.cc:482] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:48.609835 14567 raft_consensus.cc:504] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.610988 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5573b3e7e5b47f9885fc1807202d2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.614977 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "423e8d583d704fa9834026ebcd57e17e" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:48.616145 14576 raft_consensus.cc:482] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:32:48.617210 12700 leader_election.cc:341] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:48.616820 14576 raft_consensus.cc:504] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.622244 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "423e8d583d704fa9834026ebcd57e17e" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:48.624716 12698 leader_election.cc:341] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:48.625247 12698 leader_election.cc:302] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:48.626757 14567 leader_election.cc:288] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:48.627929 14567 raft_consensus.cc:2726] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:48.629721 14576 leader_election.cc:288] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:48.629935 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab8d987109ba40198645881b449cb972" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:48.631618 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab8d987109ba40198645881b449cb972" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:48.633523 12698 leader_election.cc:341] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:48.636812 12700 leader_election.cc:341] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:48.637377 12700 leader_election.cc:302] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:48.638850 14576 raft_consensus.cc:2726] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:48.647913 13283 tablet_service.cc:1467] Processing CreateTablet for tablet d5573b3e7e5b47f9885fc1807202d2e6 (DEFAULT_TABLE table=default.Table_1072812b5a8a49a1a6a1a94eb4e2b319 [id=b3a8515067ee4ce08f1d5dd8e72f56f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.649327 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5573b3e7e5b47f9885fc1807202d2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.653036 13282 tablet_service.cc:1467] Processing CreateTablet for tablet f0cba350e63e46a894a89ccc5727aab8 (DEFAULT_TABLE table=default.Table_e0909dd9de4c461dae94163efea3a62f [id=60ca2084e8394d91939269b6d596e0c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.653422 13974 ts_tablet_manager.cc:1918] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:48.653780 13974 log.cc:1196] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/874ca0c3586b41a8a3074d8988813633
I20260810 23:32:48.654275 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0cba350e63e46a894a89ccc5727aab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.654551 13974 ts_tablet_manager.cc:1939] T 874ca0c3586b41a8a3074d8988813633 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:48.676951 12766 tablet_service.cc:1467] Processing CreateTablet for tablet f0cba350e63e46a894a89ccc5727aab8 (DEFAULT_TABLE table=default.Table_e0909dd9de4c461dae94163efea3a62f [id=60ca2084e8394d91939269b6d596e0c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.682339 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0cba350e63e46a894a89ccc5727aab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.692482 14121 ts_tablet_manager.cc:1918] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:48.692916 14121 log.cc:1196] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/a0ba75f2c2b54ec1968db8feeacf8383
I20260810 23:32:48.693881 14121 ts_tablet_manager.cc:1939] T a0ba75f2c2b54ec1968db8feeacf8383 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:48.694926 13022 tablet_service.cc:1467] Processing CreateTablet for tablet f0cba350e63e46a894a89ccc5727aab8 (DEFAULT_TABLE table=default.Table_e0909dd9de4c461dae94163efea3a62f [id=60ca2084e8394d91939269b6d596e0c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.696162 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0cba350e63e46a894a89ccc5727aab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.708233 13273 tablet_service.cc:1467] Processing CreateTablet for tablet 45e39a7b58ac402da7c5e845ea6c8329 (DEFAULT_TABLE table=default.Table_4612ad4a086f41b28be40a81f4b0a41c [id=d207f6aad69f4e269854886a771760c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.709347 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 45e39a7b58ac402da7c5e845ea6c8329 (DEFAULT_TABLE table=default.Table_4612ad4a086f41b28be40a81f4b0a41c [id=d207f6aad69f4e269854886a771760c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.710521 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e39a7b58ac402da7c5e845ea6c8329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.724498 13935 ts_tablet_manager.cc:1918] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:48.725678 13935 log.cc:1196] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/874ca0c3586b41a8a3074d8988813633
I20260810 23:32:48.726603 13935 ts_tablet_manager.cc:1939] T 874ca0c3586b41a8a3074d8988813633 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:48.729143 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 163) took 5121 ms. Trace:
I20260810 23:32:48.729550 13935 rpcz_store.cc:276] 0810 23:32:43.608041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.608197 (+ 156us) service_pool.cc:227] Handling call
0810 23:32:48.258678 (+4650481us) ts_tablet_manager.cc:1145] Deleting tablet 874ca0c3586b41a8a3074d8988813633
0810 23:32:48.258788 (+ 110us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:48.539941 (+281153us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:48.724166 (+184225us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:48.729083 (+ 4917us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4564630}
I20260810 23:32:48.731073 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 874ca0c3586b41a8a3074d8988813633 (table default.Table_b57f0f6e94384d22af5d3bf7817d5044 [id=4bb9a6fbe6e644c583a468af86950cef]) successfully deleted
I20260810 23:32:48.732638 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 164) took 3759 ms. Trace:
I20260810 23:32:48.733235 13935 rpcz_store.cc:276] 0810 23:32:44.973696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:44.974275 (+ 579us) service_pool.cc:227] Handling call
0810 23:32:48.732040 (+3757765us) ts_tablet_manager.cc:1145] Deleting tablet 806fe5162aef4d36ad5e2ac324047426
0810 23:32:48.732090 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:48.732506 (+ 416us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3718140}
W20260810 23:32:48.735133 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 806fe5162aef4d36ad5e2ac324047426 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 806fe5162aef4d36ad5e2ac324047426 already in progress: creating tablet
I20260810 23:32:48.737219 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 144) took 5413 ms. Trace:
I20260810 23:32:48.737640 14121 rpcz_store.cc:276] 0810 23:32:43.324304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.324483 (+ 179us) service_pool.cc:227] Handling call
0810 23:32:48.204651 (+4880168us) ts_tablet_manager.cc:1145] Deleting tablet a0ba75f2c2b54ec1968db8feeacf8383
0810 23:32:48.204734 (+ 83us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:48.286578 (+ 81844us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b58000f1db0. stack: 00007fa86ed56033 00007fa86ed55f30 00007fa86e8d7534 00000000004db2b4 00000000004db249 00007fa875c62471 00007fa875c58f63 00007fa875cbcaba 00007fa875ce5375 00007fa8760fb26c 00007fa876d08cfa 00007fa876d38e10 00007fa876d1fe0d 00007fa876d1fdb9 00007fa876d1fd49 00007fa876d1fd11
0810 23:32:48.362464 (+ 75886us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:48.692179 (+329715us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:48.737117 (+ 44938us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38502144,"tablet-delete.queue_time_us":4731164}
I20260810 23:32:48.739727 13691 tablet_bootstrap.cc:492] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:48.740727 13273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e39a7b58ac402da7c5e845ea6c8329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.748726 13031 tablet_service.cc:1467] Processing CreateTablet for tablet 45e39a7b58ac402da7c5e845ea6c8329 (DEFAULT_TABLE table=default.Table_4612ad4a086f41b28be40a81f4b0a41c [id=d207f6aad69f4e269854886a771760c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.749552 13279 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1d3bd06a9a4627a1a7e83b22dba2af (DEFAULT_TABLE table=default.Table_764f15b83d234a90886ebfae787847ad [id=985f8cd96a404f6d8d322ed7f10f968b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.750336 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e39a7b58ac402da7c5e845ea6c8329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.751595 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1d3bd06a9a4627a1a7e83b22dba2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.756711 13691 ts_tablet_manager.cc:1397] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.267s user 0.013s sys 0.011s
I20260810 23:32:48.767071 13935 tablet_replica.cc:330] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:48.772368 13935 raft_consensus.cc:2229] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:48.777222 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 166) took 4539 ms. Trace:
I20260810 23:32:48.777634 13974 rpcz_store.cc:276] 0810 23:32:44.237325 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:44.237937 (+ 612us) service_pool.cc:227] Handling call
0810 23:32:48.221756 (+3983819us) ts_tablet_manager.cc:1145] Deleting tablet 874ca0c3586b41a8a3074d8988813633
0810 23:32:48.221803 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:48.513989 (+292186us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:48.653144 (+139155us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:48.777132 (+123988us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7775,"tablet-delete.queue_time_us":3980268}
I20260810 23:32:48.779166 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 874ca0c3586b41a8a3074d8988813633 (table default.Table_b57f0f6e94384d22af5d3bf7817d5044 [id=4bb9a6fbe6e644c583a468af86950cef]) successfully deleted
I20260810 23:32:48.779654 13974 tablet_replica.cc:330] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:48.780726 13974 raft_consensus.cc:2229] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:48.780788 14121 tablet_replica.cc:330] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:48.780956 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet a0ba75f2c2b54ec1968db8feeacf8383 (table default.Table_dd2cd30e5e2349de9180be01866dc513 [id=8754da7be72b4eef80daacdcdfde82a7]) successfully deleted
I20260810 23:32:48.781384 13974 raft_consensus.cc:2258] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:48.781731 14121 raft_consensus.cc:2229] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:48.782364 14121 raft_consensus.cc:2258] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:48.767345 13691 raft_consensus.cc:348] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.782843 13691 raft_consensus.cc:374] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:48.783406 13974 ts_tablet_manager.cc:1905] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:48.783732 13691 raft_consensus.cc:729] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:48.787276 13691 consensus_queue.cc:260] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:48.797581 13691 ts_tablet_manager.cc:1428] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.040s user 0.006s sys 0.002s
I20260810 23:32:48.799557 13665 tablet_bootstrap.cc:654] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:48.803874 13007 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1d3bd06a9a4627a1a7e83b22dba2af (DEFAULT_TABLE table=default.Table_764f15b83d234a90886ebfae787847ad [id=985f8cd96a404f6d8d322ed7f10f968b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.805939 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 95d18289e2ae4306b4699a6d83f452a7 (DEFAULT_TABLE table=default.Table_69797cc483424ae99a99731b36130f0f [id=82e5367dce5143c7b396b145a3faccc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.811725 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff43927ed204bfda6f647808b44fa3a (DEFAULT_TABLE table=default.Table_6c7fe24265af4acaab757c9857a87a7e [id=e539d34e2b7c48bebe9e2729946b3856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.817407 12224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 62705f99687340aaa1aca0942b28487f
I20260810 23:32:48.824190 13691 tablet_bootstrap.cc:492] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:48.824534 13278 tablet_service.cc:1514] Processing DeleteTablet for tablet 62705f99687340aaa1aca0942b28487f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:48 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:48.825538 14121 ts_tablet_manager.cc:1905] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:48.828100 13935 raft_consensus.cc:2258] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:48.830175 13935 ts_tablet_manager.cc:1905] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:48.834338 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 62705f99687340aaa1aca0942b28487f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:48 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:48.843436 13007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1d3bd06a9a4627a1a7e83b22dba2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.844560 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d18289e2ae4306b4699a6d83f452a7. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:48.846031 12952 consensus_peers.cc:487] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 874ca0c3586b41a8a3074d8988813633. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:48.848616 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff43927ed204bfda6f647808b44fa3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.819689 12224 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 258) took 2471 ms. Trace:
I20260810 23:32:48.850867 12224 rpcz_store.cc:276] 0810 23:32:46.348219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:48.038881 (+1690662us) service_pool.cc:227] Handling call
0810 23:32:48.041387 (+ 2506us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:48.043105 (+ 1718us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:48.043280 (+ 175us) catalog_manager.cc:2761] Locking tablets
0810 23:32:48.387483 (+344203us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:48.816839 (+429356us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:48.817126 (+ 287us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:48.817278 (+ 152us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:48.819598 (+ 2320us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:48.390021 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:48.390500 (+ 479us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:48.390530 (+ 30us) write_op.cc:435] Acquired schema lock
0810 23:32:48.390546 (+ 16us) tablet.cc:660] Decoding operations
0810 23:32:48.390949 (+ 403us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:48.391012 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:48.391042 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:48.391399 (+ 357us) tablet.cc:699] Row locks acquired
0810 23:32:48.391420 (+ 21us) write_op.cc:260] PREPARE: finished
0810 23:32:48.391587 (+ 167us) write_op.cc:270] Start()
0810 23:32:48.391707 (+ 120us) write_op.cc:276] Timestamp: P: 1786404768391527 usec, L: 0
0810 23:32:48.391727 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:32:48.424213 (+ 32486us) log.cc:842] Serialized 734 byte log entry
0810 23:32:48.426321 (+ 2108us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:48.810760 (+384439us) op_driver.cc:444] REPLICATION: finished
0810 23:32:48.811939 (+ 1179us) write_op.cc:301] APPLY: starting
0810 23:32:48.812071 (+ 132us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:48.812310 (+ 239us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:48.812359 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:48.812911 (+ 552us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:48.813244 (+ 333us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:48.813348 (+ 104us) write_op.cc:312] APPLY: finished
0810 23:32:48.813670 (+ 322us) log.cc:842] Serialized 29 byte log entry
0810 23:32:48.813911 (+ 241us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:48.814010 (+ 99us) write_op.cc:454] Released schema lock
0810 23:32:48.814054 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:48.814348 (+ 294us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:48.814385 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":882,"apply.run_cpu_time_us":2704,"apply.run_wall_time_us":2704,"num_ops":2,"prepare.queue_time_us":308,"prepare.run_cpu_time_us":4046,"prepare.run_wall_time_us":36455,"replication_time_us":418865,"spinlock_wait_cycles":72689920}]]}
I20260810 23:32:48.861871 13023 tablet_service.cc:1467] Processing CreateTablet for tablet 95d18289e2ae4306b4699a6d83f452a7 (DEFAULT_TABLE table=default.Table_69797cc483424ae99a99731b36130f0f [id=82e5367dce5143c7b396b145a3faccc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.863344 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d18289e2ae4306b4699a6d83f452a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.873315 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 95d18289e2ae4306b4699a6d83f452a7 (DEFAULT_TABLE table=default.Table_69797cc483424ae99a99731b36130f0f [id=82e5367dce5143c7b396b145a3faccc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.875561 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet 62705f99687340aaa1aca0942b28487f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:48 UTC) from {username='slave'} at 127.0.0.1:47546
W20260810 23:32:48.879196 12958 consensus_peers.cc:487] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:48.889020 12763 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1d3bd06a9a4627a1a7e83b22dba2af (DEFAULT_TABLE table=default.Table_764f15b83d234a90886ebfae787847ad [id=985f8cd96a404f6d8d322ed7f10f968b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.875554 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d18289e2ae4306b4699a6d83f452a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.883476 12226 catalog_manager.cc:5495] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f26517150f1c480a877f15aec71d020b (127.11.226.130). New cstate: current_term: 1 leader_uuid: "f26517150f1c480a877f15aec71d020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } }
I20260810 23:32:48.900683 13691 tablet_bootstrap.cc:654] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:48.901309 12226 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 6a3a834d77474c3d8feaa5a01f3faff3 (table default.Table_dfb60e6e40874a5da0c56f6926ce1f63 [id=1ea3dfab929845f89d4680129ae45d94]). Expected version 1 got 0
I20260810 23:32:48.902438 12226 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet bd77526f1db4455291fde8a001092a09 (table default.Table_ac23e0995fcd414ead1b88406cf7d79e [id=963512beef0d4063b8e6a5738abd1d92]). Expected version 2 got 0
I20260810 23:32:48.909969 12226 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
I20260810 23:32:48.912976 13008 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff43927ed204bfda6f647808b44fa3a (DEFAULT_TABLE table=default.Table_6c7fe24265af4acaab757c9857a87a7e [id=e539d34e2b7c48bebe9e2729946b3856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.925312 13008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff43927ed204bfda6f647808b44fa3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.937211 13282 tablet_service.cc:1467] Processing CreateTablet for tablet dc7adcb46ae0472384b3bb883ea58a74 (DEFAULT_TABLE table=default.Table_1ec6152b7f994da6987cb0683216ade9 [id=6b7c9683647d4c2e94418e9f2dcaa41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.938846 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7adcb46ae0472384b3bb883ea58a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.961314 12755 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff43927ed204bfda6f647808b44fa3a (DEFAULT_TABLE table=default.Table_6c7fe24265af4acaab757c9857a87a7e [id=e539d34e2b7c48bebe9e2729946b3856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.964505 13665 tablet_bootstrap.cc:492] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:48.964802 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1d3bd06a9a4627a1a7e83b22dba2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.966724 12764 tablet_service.cc:1467] Processing CreateTablet for tablet dc7adcb46ae0472384b3bb883ea58a74 (DEFAULT_TABLE table=default.Table_1ec6152b7f994da6987cb0683216ade9 [id=6b7c9683647d4c2e94418e9f2dcaa41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.968407 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7adcb46ae0472384b3bb883ea58a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.988974 12769 tablet_service.cc:1467] Processing CreateTablet for tablet d01f1ec3c5e945378fdd008ba13dceaa (DEFAULT_TABLE table=default.Table_a7072962772044ad8f3e9a8e446baea1 [id=103311fa8f874c1a9ce05417581d7483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.990649 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff43927ed204bfda6f647808b44fa3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.993373 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01f1ec3c5e945378fdd008ba13dceaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:48.996578 13665 ts_tablet_manager.cc:1397] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.400s user 0.013s sys 0.008s
I20260810 23:32:48.988242 13276 tablet_service.cc:1467] Processing CreateTablet for tablet d01f1ec3c5e945378fdd008ba13dceaa (DEFAULT_TABLE table=default.Table_a7072962772044ad8f3e9a8e446baea1 [id=103311fa8f874c1a9ce05417581d7483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:48.991340 14582 raft_consensus.cc:482] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:49.000680 14582 raft_consensus.cc:504] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.005524 14582 leader_election.cc:288] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:49.007174 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01f1ec3c5e945378fdd008ba13dceaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.011207 13514 tablet_bootstrap.cc:492] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:49.011680 13514 ts_tablet_manager.cc:1397] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.537s user 0.013s sys 0.011s
I20260810 23:32:49.032898 13278 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4c2810b36f4e7a93443bdc1a07485a (DEFAULT_TABLE table=default.Table_c5dbbcbf87314046b7357e16f8b94e35 [id=0cbb4cd2d3f94123a65ebc79cd54a870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.038909 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4c2810b36f4e7a93443bdc1a07485a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.040932 12229 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet bd77526f1db4455291fde8a001092a09 (table default.Table_ac23e0995fcd414ead1b88406cf7d79e [id=963512beef0d4063b8e6a5738abd1d92]). Expected version 2 got 0
I20260810 23:32:49.042102 12229 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
I20260810 23:32:48.999895 13665 raft_consensus.cc:348] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.042579 13665 raft_consensus.cc:374] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.042870 13665 raft_consensus.cc:729] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.051151 12237 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet bd77526f1db4455291fde8a001092a09 (table default.Table_ac23e0995fcd414ead1b88406cf7d79e [id=963512beef0d4063b8e6a5738abd1d92]). Expected version 2 got 0
I20260810 23:32:49.054811 12237 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:49.054939 13665 consensus_queue.cc:260] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.056360 12237 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
I20260810 23:32:49.057221 12237 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 6a3a834d77474c3d8feaa5a01f3faff3 (table default.Table_dfb60e6e40874a5da0c56f6926ce1f63 [id=1ea3dfab929845f89d4680129ae45d94]). Expected version 1 got 0
I20260810 23:32:49.069929 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
I20260810 23:32:49.071290 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet bd77526f1db4455291fde8a001092a09 (table default.Table_ac23e0995fcd414ead1b88406cf7d79e [id=963512beef0d4063b8e6a5738abd1d92]). Expected version 2 got 0
I20260810 23:32:49.078658 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4c2810b36f4e7a93443bdc1a07485a (DEFAULT_TABLE table=default.Table_c5dbbcbf87314046b7357e16f8b94e35 [id=0cbb4cd2d3f94123a65ebc79cd54a870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.094961 13665 ts_tablet_manager.cc:1428] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.098s user 0.007s sys 0.001s
I20260810 23:32:49.096351 13665 tablet_bootstrap.cc:492] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:49.097962 12224 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
I20260810 23:32:48.933748 13691 tablet_bootstrap.cc:492] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:49.110005 13691 ts_tablet_manager.cc:1397] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.286s user 0.009s sys 0.009s
I20260810 23:32:49.103760 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42517f11422b42c39db282e2c5f584a3" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:49.105597 13031 tablet_service.cc:1467] Processing CreateTablet for tablet dc7adcb46ae0472384b3bb883ea58a74 (DEFAULT_TABLE table=default.Table_1ec6152b7f994da6987cb0683216ade9 [id=6b7c9683647d4c2e94418e9f2dcaa41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.117885 12223 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet bd77526f1db4455291fde8a001092a09 (table default.Table_ac23e0995fcd414ead1b88406cf7d79e [id=963512beef0d4063b8e6a5738abd1d92]). Expected version 2 got 0
I20260810 23:32:49.119342 12223 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:49.120775 12223 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
I20260810 23:32:49.122244 12223 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 6a3a834d77474c3d8feaa5a01f3faff3 (table default.Table_dfb60e6e40874a5da0c56f6926ce1f63 [id=1ea3dfab929845f89d4680129ae45d94]). Expected version 1 got 0
I20260810 23:32:49.145210 13062 raft_consensus.cc:2454] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:49.150574 12240 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_d0b8422dca5e40ff919741fcee7d547c [id=71c9424f327040a68889fc257935c535] alter complete (version 3)
W20260810 23:32:49.157389 12952 consensus_peers.cc:487] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:49.158677 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7adcb46ae0472384b3bb883ea58a74. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:49.158663 12236 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 23) took 3988 ms (client timeout 1000 ms). Trace:
I20260810 23:32:49.160355 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4c2810b36f4e7a93443bdc1a07485a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.166082 13691 raft_consensus.cc:348] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:49.166815 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 124259164adf404c9cbe5f4e0d5e16c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:49.167074 13691 raft_consensus.cc:374] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.167325 13691 raft_consensus.cc:729] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.170643 14567 consensus_queue.cc:1035] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:49.172320 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 874ca0c3586b41a8a3074d8988813633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:49.173079 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ad5d63a2104995990a86a000963cdc with delete_type TABLET_DATA_DELETED (replaced by tablet 8fb2ee1c9b41478da001aa67d7a7ad3c) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:49.177683 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 440de7d9de8d4be9a5322e1fe3388b9b with delete_type TABLET_DATA_DELETED (replaced by tablet e62dc27b59ba469d99e8d11372aceff5) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:49.178539 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet db363eb0f98d460eb6f5a577233d8ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:36 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:49.167997 13691 consensus_queue.cc:260] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:49.184127 13691 ts_tablet_manager.cc:1428] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.074s user 0.003s sys 0.007s
I20260810 23:32:49.187726 13935 ts_tablet_manager.cc:1918] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260810 23:32:49.188727 12228 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 51) took 2670 ms (client timeout 1000 ms). Trace:
I20260810 23:32:49.149130 13212 leader_election.cc:302] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:49.191156 14582 raft_consensus.cc:2781] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:49.191542 14582 raft_consensus.cc:482] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:49.191867 14582 raft_consensus.cc:3037] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:49.196131 13213 leader_election.cc:341] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 42517f11422b42c39db282e2c5f584a3
W20260810 23:32:49.199203 12228 rpcz_store.cc:269] 0810 23:32:46.518677 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:47.123378 (+604701us) service_pool.cc:227] Handling call
0810 23:32:49.188618 (+2065240us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:48.426635 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:48.427159 (+ 524us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:48.427187 (+ 28us) write_op.cc:435] Acquired schema lock
0810 23:32:48.427205 (+ 18us) tablet.cc:660] Decoding operations
0810 23:32:48.427816 (+ 611us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:48.427862 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:48.427885 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:48.428096 (+ 211us) tablet.cc:699] Row locks acquired
0810 23:32:48.428109 (+ 13us) write_op.cc:260] PREPARE: finished
0810 23:32:48.428249 (+ 140us) write_op.cc:270] Start()
0810 23:32:48.428350 (+ 101us) write_op.cc:276] Timestamp: P: 1786404768428202 usec, L: 0
0810 23:32:48.428370 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:32:48.428929 (+ 559us) log.cc:842] Serialized 536 byte log entry
0810 23:32:48.811191 (+382262us) op_driver.cc:444] REPLICATION: finished
0810 23:32:48.814894 (+ 3703us) write_op.cc:301] APPLY: starting
0810 23:32:48.814973 (+ 79us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:48.815131 (+ 158us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:48.815273 (+ 142us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:48.815580 (+ 307us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:48.815789 (+ 209us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:48.815849 (+ 60us) write_op.cc:312] APPLY: finished
0810 23:32:48.863015 (+ 47166us) log.cc:842] Serialized 23 byte log entry
0810 23:32:48.864766 (+ 1751us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:48.864921 (+ 155us) write_op.cc:454] Released schema lock
0810 23:32:48.865005 (+ 84us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:48.895607 (+ 30602us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:48.895640 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":3482,"apply.run_cpu_time_us":2725,"apply.run_wall_time_us":81063,"num_ops":1,"prepare.queue_time_us":6129,"prepare.run_cpu_time_us":2326,"prepare.run_wall_time_us":3623,"replication_time_us":382685,"thread_start_us":83301,"threads_started":1}]]}
W20260810 23:32:49.193728 12234 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 50) took 3802 ms (client timeout 1000 ms). Trace:
I20260810 23:32:49.204362 13278 tablet_service.cc:1514] Processing DeleteTablet for tablet fd110664bac34f52bfe922a3a76c6290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:31 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:49.205740 14582 raft_consensus.cc:504] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:49.203509 12234 rpcz_store.cc:269] 0810 23:32:45.390746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.408861 (+ 18115us) service_pool.cc:227] Handling call
0810 23:32:45.462073 (+ 53212us) spinlock_profiling.cc:243] Waited 50.6 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:49.193504 (+3731431us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:47.404764 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:47.405589 (+ 825us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:47.405641 (+ 52us) write_op.cc:435] Acquired schema lock
0810 23:32:47.405679 (+ 38us) tablet.cc:660] Decoding operations
0810 23:32:47.406315 (+ 636us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:47.406390 (+ 75us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:47.406445 (+ 55us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:47.406758 (+ 313us) tablet.cc:699] Row locks acquired
0810 23:32:47.406798 (+ 40us) write_op.cc:260] PREPARE: finished
0810 23:32:47.407094 (+ 296us) write_op.cc:270] Start()
0810 23:32:47.407210 (+ 116us) write_op.cc:276] Timestamp: P: 1786404767407028 usec, L: 0
0810 23:32:47.407231 (+ 21us) op_driver.cc:331] REPLICATION: starting
0810 23:32:47.455183 (+ 47952us) log.cc:842] Serialized 536 byte log entry
0810 23:32:47.727784 (+272601us) op_driver.cc:444] REPLICATION: finished
0810 23:32:47.731236 (+ 3452us) write_op.cc:301] APPLY: starting
0810 23:32:47.731336 (+ 100us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:47.731482 (+ 146us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:47.731599 (+ 117us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:47.732081 (+ 482us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:47.732318 (+ 237us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:47.732423 (+ 105us) write_op.cc:312] APPLY: finished
0810 23:32:47.732771 (+ 348us) log.cc:842] Serialized 23 byte log entry
0810 23:32:47.735000 (+ 2229us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:47.735115 (+ 115us) write_op.cc:454] Released schema lock
0810 23:32:47.735180 (+ 65us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:47.745543 (+ 10363us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:47.745610 (+ 67us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":8,"spinlock_wait_cycles":116365696,"child_traces":[["op",{"apply.queue_time_us":3193,"apply.run_cpu_time_us":2912,"apply.run_wall_time_us":14755,"num_ops":1,"prepare.queue_time_us":432738,"prepare.run_cpu_time_us":4229,"prepare.run_wall_time_us":60375,"replication_time_us":320400}]]}
I20260810 23:32:49.199054 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet 133654008f5247b2ac87dcd09195f197 with delete_type TABLET_DATA_DELETED (replaced by tablet ca8c0d1dacc344cbacb0c5988f0e2221) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:49.199694 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet b261673faa964b65af3576c89e302864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:49.202545 12766 tablet_service.cc:1514] Processing DeleteTablet for tablet b261673faa964b65af3576c89e302864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:49.185667 13691 tablet_bootstrap.cc:492] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:49.208374 13935 log.cc:1196] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/4d697e7533af4fc0a15cb41f3fcce4db
I20260810 23:32:49.210201 13935 ts_tablet_manager.cc:1939] T 4d697e7533af4fc0a15cb41f3fcce4db P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
W20260810 23:32:49.211784 12226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 52) took 1418 ms (client timeout 1000 ms). Trace:
W20260810 23:32:49.197340 12240 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 31) took 3869 ms (client timeout 1000 ms). Trace:
W20260810 23:32:49.159039 12236 rpcz_store.cc:269] 0810 23:32:45.170258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.170442 (+ 184us) service_pool.cc:227] Handling call
0810 23:32:49.158566 (+3988124us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:45.659191 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:45.660041 (+ 850us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:45.660083 (+ 42us) write_op.cc:435] Acquired schema lock
0810 23:32:45.660111 (+ 28us) tablet.cc:660] Decoding operations
0810 23:32:45.660727 (+ 616us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:45.660806 (+ 79us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:45.660857 (+ 51us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:45.661167 (+ 310us) tablet.cc:699] Row locks acquired
0810 23:32:45.661195 (+ 28us) write_op.cc:260] PREPARE: finished
0810 23:32:45.661403 (+ 208us) write_op.cc:270] Start()
0810 23:32:45.661550 (+ 147us) write_op.cc:276] Timestamp: P: 1786404765661305 usec, L: 0
0810 23:32:45.661582 (+ 32us) op_driver.cc:331] REPLICATION: starting
0810 23:32:45.662415 (+ 833us) log.cc:842] Serialized 356 byte log entry
0810 23:32:45.677095 (+ 14680us) op_driver.cc:444] REPLICATION: finished
0810 23:32:45.677819 (+ 724us) write_op.cc:301] APPLY: starting
0810 23:32:45.677923 (+ 104us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:45.678107 (+ 184us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:45.678277 (+ 170us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:45.678614 (+ 337us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:45.678910 (+ 296us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:45.679001 (+ 91us) write_op.cc:312] APPLY: finished
0810 23:32:45.679314 (+ 313us) log.cc:842] Serialized 23 byte log entry
0810 23:32:45.679566 (+ 252us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:45.679692 (+ 126us) write_op.cc:454] Released schema lock
0810 23:32:45.679749 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:45.682600 (+ 2851us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:45.682632 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":453,"apply.run_cpu_time_us":2535,"apply.run_wall_time_us":5020,"num_ops":1,"prepare.queue_time_us":249680,"prepare.run_cpu_time_us":3727,"prepare.run_wall_time_us":28082,"raft.queue_time_us":101,"raft.run_cpu_time_us":872,"raft.run_wall_time_us":872,"replication_time_us":15373}]]}
I20260810 23:32:49.205355 13514 raft_consensus.cc:348] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.221103 13514 raft_consensus.cc:374] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.221402 13514 raft_consensus.cc:729] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.222123 13514 consensus_queue.cc:260] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.206703 12757 tablet_service.cc:1467] Processing CreateTablet for tablet dfda619f19a5459494b1e4f6d22821cc (DEFAULT_TABLE table=default.Table_77274019ccb14c57b2a9ce882ee84dac [id=7184e7fca1b54e72a7ba15d5453baa37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.223845 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfda619f19a5459494b1e4f6d22821cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.210608 13007 tablet_service.cc:1467] Processing CreateTablet for tablet d01f1ec3c5e945378fdd008ba13dceaa (DEFAULT_TABLE table=default.Table_a7072962772044ad8f3e9a8e446baea1 [id=103311fa8f874c1a9ce05417581d7483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.237068 13007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01f1ec3c5e945378fdd008ba13dceaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.238251 14582 leader_election.cc:288] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:49.244566 12228 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_ca9eee6f19c34490b6a9cad315a8f740" } new_table_name: "default.Table_5cde9c2beb0448c1971328536e2c47d8" modify_external_catalogs: true
W20260810 23:32:49.216355 12240 rpcz_store.cc:269] 0810 23:32:45.327893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.328990 (+ 1097us) service_pool.cc:227] Handling call
0810 23:32:45.396918 (+ 67928us) spinlock_profiling.cc:243] Waited 59.2 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:49.197253 (+3800335us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:47.465378 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:47.469468 (+ 4090us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:47.469534 (+ 66us) write_op.cc:435] Acquired schema lock
0810 23:32:47.469546 (+ 12us) tablet.cc:660] Decoding operations
0810 23:32:47.472711 (+ 3165us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:47.472764 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:47.472825 (+ 61us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:47.473114 (+ 289us) tablet.cc:699] Row locks acquired
0810 23:32:47.473154 (+ 40us) write_op.cc:260] PREPARE: finished
0810 23:32:47.473391 (+ 237us) write_op.cc:270] Start()
0810 23:32:47.473569 (+ 178us) write_op.cc:276] Timestamp: P: 1786404767473317 usec, L: 0
0810 23:32:47.473586 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:32:47.474773 (+ 1187us) log.cc:842] Serialized 356 byte log entry
0810 23:32:47.730380 (+255607us) op_driver.cc:444] REPLICATION: finished
0810 23:32:47.747750 (+ 17370us) write_op.cc:301] APPLY: starting
0810 23:32:47.749547 (+ 1797us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:47.749680 (+ 133us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:47.749890 (+ 210us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:47.750162 (+ 272us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:47.750439 (+ 277us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:47.750538 (+ 99us) write_op.cc:312] APPLY: finished
0810 23:32:47.764799 (+ 14261us) log.cc:842] Serialized 23 byte log entry
0810 23:32:47.774040 (+ 9241us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:47.774158 (+ 118us) write_op.cc:454] Released schema lock
0810 23:32:47.774205 (+ 47us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:47.780453 (+ 6248us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:47.780501 (+ 48us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":25984,"reported_tablets":16,"spinlock_wait_cycles":136161024,"child_traces":[["op",{"apply.queue_time_us":15667,"apply.run_cpu_time_us":2944,"apply.run_wall_time_us":33074,"num_ops":1,"prepare.queue_time_us":322721,"prepare.run_cpu_time_us":4531,"prepare.run_wall_time_us":167825,"raft.queue_time_us":254,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":65,"replication_time_us":256608,"thread_start_us":101923,"threads_started":1}]]}
W20260810 23:32:49.254176 12698 consensus_peers.cc:487] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:49.225561 12224 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_ac23e0995fcd414ead1b88406cf7d79e [id=963512beef0d4063b8e6a5738abd1d92] alter complete (version 2)
I20260810 23:32:49.258849 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 165) took 4146 ms. Trace:
I20260810 23:32:49.259331 13935 rpcz_store.cc:276] 0810 23:32:45.112947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.114247 (+ 1300us) service_pool.cc:227] Handling call
0810 23:32:48.736759 (+3622512us) ts_tablet_manager.cc:1145] Deleting tablet 4d697e7533af4fc0a15cb41f3fcce4db
0810 23:32:48.766863 (+ 30104us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:48.915995 (+149132us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:49.187385 (+271390us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:49.258772 (+ 71387us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98082,"tablet-delete.queue_time_us":3621261}
I20260810 23:32:49.247030 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42517f11422b42c39db282e2c5f584a3" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:49.261147 13062 raft_consensus.cc:3037] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.263329 13277 tablet_service.cc:1514] Processing DeleteTablet for tablet 124259164adf404c9cbe5f4e0d5e16c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:49.264446 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 081b4e1f53c74f768a4bf401ae22b585 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:32 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:49.264926 12235 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 25) took 1851 ms (client timeout 1000 ms). Trace:
I20260810 23:32:49.265353 13278 tablet_service.cc:1514] Processing DeleteTablet for tablet b261673faa964b65af3576c89e302864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:49.267203 12235 rpcz_store.cc:269] 0810 23:32:47.413636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:47.603641 (+190005us) service_pool.cc:227] Handling call
0810 23:32:47.650336 (+ 46695us) spinlock_profiling.cc:243] Waited 28 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:49.264823 (+1614487us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":8,"spinlock_wait_cycles":64285440}
I20260810 23:32:49.270372 13514 ts_tablet_manager.cc:1428] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.258s user 0.006s sys 0.002s
I20260810 23:32:49.271718 12235 master_service.cc:946] ReplaceTablet: received request to replace tablet 4d697e7533af4fc0a15cb41f3fcce4db from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:49.274479 12235 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 260) took 1983 ms. Trace:
W20260810 23:32:49.275473 13213 leader_election.cc:341] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 42517f11422b42c39db282e2c5f584a3
I20260810 23:32:49.276870 13514 tablet_bootstrap.cc:492] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:49.280900 13278 tablet_service.cc:1514] Processing DeleteTablet for tablet 133654008f5247b2ac87dcd09195f197 with delete_type TABLET_DATA_DELETED (replaced by tablet ca8c0d1dacc344cbacb0c5988f0e2221) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:49.282200 13935 tablet_replica.cc:330] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:49.283146 13935 raft_consensus.cc:2229] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:49.286152 13935 raft_consensus.cc:2258] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:49.290067 13665 tablet_bootstrap.cc:654] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.290819 12235 rpcz_store.cc:276] 0810 23:32:47.291165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.271564 (+1980399us) service_pool.cc:227] Handling call
0810 23:32:49.274391 (+ 2827us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 23:32:49.258894 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 136268us
W20260810 23:32:49.295359 12229 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 24) took 2956 ms (client timeout 1000 ms). Trace:
W20260810 23:32:49.296233 12229 rpcz_store.cc:269] 0810 23:32:46.339171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:46.977520 (+638349us) service_pool.cc:227] Handling call
0810 23:32:46.998750 (+ 21230us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:49.295278 (+2296528us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":7,"spinlock_wait_cycles":39032064}
I20260810 23:32:49.297240 13022 tablet_service.cc:1467] Processing CreateTablet for tablet dfda619f19a5459494b1e4f6d22821cc (DEFAULT_TABLE table=default.Table_77274019ccb14c57b2a9ce882ee84dac [id=7184e7fca1b54e72a7ba15d5453baa37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.297510 13014 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ba75f2c2b54ec1968db8feeacf8383 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c29b11192604ac18c21b7bdc0238fb3) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:49.212648 12226 rpcz_store.cc:269] 0810 23:32:47.793073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:47.906416 (+113343us) service_pool.cc:227] Handling call
0810 23:32:47.962344 (+ 55928us) spinlock_profiling.cc:243] Waited 52.6 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:49.211652 (+1249308us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:49.005667 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:49.006448 (+ 781us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:49.006508 (+ 60us) write_op.cc:435] Acquired schema lock
0810 23:32:49.006532 (+ 24us) tablet.cc:660] Decoding operations
0810 23:32:49.007087 (+ 555us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:49.007196 (+ 109us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:49.007282 (+ 86us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:49.007608 (+ 326us) tablet.cc:699] Row locks acquired
0810 23:32:49.007633 (+ 25us) write_op.cc:260] PREPARE: finished
0810 23:32:49.007847 (+ 214us) write_op.cc:270] Start()
0810 23:32:49.014934 (+ 7087us) write_op.cc:276] Timestamp: P: 1786404769007803 usec, L: 0
0810 23:32:49.014957 (+ 23us) op_driver.cc:331] REPLICATION: starting
0810 23:32:49.015928 (+ 971us) log.cc:842] Serialized 536 byte log entry
0810 23:32:49.030679 (+ 14751us) op_driver.cc:444] REPLICATION: finished
0810 23:32:49.033964 (+ 3285us) write_op.cc:301] APPLY: starting
0810 23:32:49.034039 (+ 75us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:49.034140 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:49.034264 (+ 124us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:49.034603 (+ 339us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:49.034863 (+ 260us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:49.034945 (+ 82us) write_op.cc:312] APPLY: finished
0810 23:32:49.035256 (+ 311us) log.cc:842] Serialized 23 byte log entry
0810 23:32:49.035596 (+ 340us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.035700 (+ 104us) write_op.cc:454] Released schema lock
0810 23:32:49.035754 (+ 54us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:49.035980 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.036013 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":13,"spinlock_wait_cycles":121036160,"child_traces":[["op",{"apply.queue_time_us":3003,"apply.run_cpu_time_us":2287,"apply.run_wall_time_us":4108,"num_ops":1,"prepare.queue_time_us":49654,"prepare.run_cpu_time_us":3915,"prepare.run_wall_time_us":31465,"raft.queue_time_us":111,"raft.run_cpu_time_us":1084,"raft.run_wall_time_us":1087,"replication_time_us":15589,"thread_start_us":58873,"threads_started":1}]]}
I20260810 23:32:49.297765 12229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_75ca982c15e34c8d9603f100dff5b392" } modify_external_catalogs: true
I20260810 23:32:49.296644 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4c2810b36f4e7a93443bdc1a07485a (DEFAULT_TABLE table=default.Table_c5dbbcbf87314046b7357e16f8b94e35 [id=0cbb4cd2d3f94123a65ebc79cd54a870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.298337 12229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_75ca982c15e34c8d9603f100dff5b392" } modify_external_catalogs: true
I20260810 23:32:49.298532 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfda619f19a5459494b1e4f6d22821cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.298686 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_3b3f1f30941b41d1a42bad9834394fe9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:49.299321 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4c2810b36f4e7a93443bdc1a07485a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.298554 13017 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b29cda94584badb1225477ec2bd1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:33 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:49.285462 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 4d697e7533af4fc0a15cb41f3fcce4db (table default.Table_26bf2c7d86704b5b981ecebb4e90ba75 [id=cefe7b97c1db42ef8712bf3c7a359af2]) successfully deleted
I20260810 23:32:49.275668 12240 master_service.cc:946] ReplaceTablet: received request to replace tablet 240adfa4be844518bbc9624d0e408789 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:49.308171 13017 tablet_service.cc:1514] Processing DeleteTablet for tablet 124259164adf404c9cbe5f4e0d5e16c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:49.308847 13014 tablet_service.cc:1514] Processing DeleteTablet for tablet b8986e59b2314bd69b9b1de36f1f2ba7 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d697e7533af4fc0a15cb41f3fcce4db) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:49.310441 13014 tablet_service.cc:1514] Processing DeleteTablet for tablet b261673faa964b65af3576c89e302864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:49.312127 13014 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d596299c82b4ff1ac151d68b97ff55b with delete_type TABLET_DATA_DELETED (replaced by tablet 4cdec13ee70d402fa3a08119e7bf1313) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:49.312726 12224 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 26) took 809 ms (client timeout 1000 ms). Trace:
I20260810 23:32:49.312832 13278 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ad5d63a2104995990a86a000963cdc with delete_type TABLET_DATA_DELETED (replaced by tablet 8fb2ee1c9b41478da001aa67d7a7ad3c) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:49.314424 13691 tablet_bootstrap.cc:654] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.220108 13283 tablet_service.cc:1467] Processing CreateTablet for tablet dfda619f19a5459494b1e4f6d22821cc (DEFAULT_TABLE table=default.Table_77274019ccb14c57b2a9ce882ee84dac [id=7184e7fca1b54e72a7ba15d5453baa37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.318320 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfda619f19a5459494b1e4f6d22821cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.318651 13014 tablet_service.cc:1514] Processing DeleteTablet for tablet 081b4e1f53c74f768a4bf401ae22b585 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:32 UTC) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:49.300398 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b3f1f30941b41d1a42bad9834394fe9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:49.327651 13935 ts_tablet_manager.cc:1905] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:49.330194 12952 consensus_peers.cc:487] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:49.317018 12235 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_f34283baaf0d495b8a5b076e45292979" } new_table_name: "default.Table_5e75f4eb90d9446d89a8395b2aa65a1d" modify_external_catalogs: true
W20260810 23:32:49.331581 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8fb2ee1c9b41478da001aa67d7a7ad3c consensus state has no leader
W20260810 23:32:49.321624 12224 rpcz_store.cc:269] 0810 23:32:48.503243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:48.904711 (+401468us) service_pool.cc:227] Handling call
0810 23:32:49.145980 (+241269us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.146009 (+ 29us) write_op.cc:454] Released schema lock
0810 23:32:49.312643 (+166634us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:49.123983 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:49.124500 (+ 517us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:49.124530 (+ 30us) write_op.cc:435] Acquired schema lock
0810 23:32:49.124546 (+ 16us) tablet.cc:660] Decoding operations
0810 23:32:49.124889 (+ 343us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:49.124943 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:49.124972 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:49.125154 (+ 182us) tablet.cc:699] Row locks acquired
0810 23:32:49.125166 (+ 12us) write_op.cc:260] PREPARE: finished
0810 23:32:49.125320 (+ 154us) write_op.cc:270] Start()
0810 23:32:49.125426 (+ 106us) write_op.cc:276] Timestamp: P: 1786404769125259 usec, L: 0
0810 23:32:49.125445 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 23:32:49.126110 (+ 665us) log.cc:842] Serialized 356 byte log entry
0810 23:32:49.135951 (+ 9841us) op_driver.cc:444] REPLICATION: finished
0810 23:32:49.136386 (+ 435us) write_op.cc:301] APPLY: starting
0810 23:32:49.136473 (+ 87us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:49.136600 (+ 127us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:49.136724 (+ 124us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:49.136901 (+ 177us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:49.137042 (+ 141us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:49.137080 (+ 38us) write_op.cc:312] APPLY: finished
0810 23:32:49.137280 (+ 200us) log.cc:842] Serialized 23 byte log entry
0810 23:32:49.137446 (+ 166us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.137518 (+ 72us) write_op.cc:454] Released schema lock
0810 23:32:49.137547 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":32143,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":240,"apply.run_cpu_time_us":1358,"apply.run_wall_time_us":1359,"num_ops":1,"prepare.queue_time_us":6326,"prepare.run_cpu_time_us":2466,"prepare.run_wall_time_us":2465,"raft.queue_time_us":1325,"raft.run_cpu_time_us":700,"raft.run_wall_time_us":18148,"replication_time_us":10419}]]}
I20260810 23:32:49.354079 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_00897e6950fb4847a8b9d4c8f709cc50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:49.356611 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00897e6950fb4847a8b9d4c8f709cc50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:49.356904 13514 tablet_bootstrap.cc:654] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.347450 12236 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
I20260810 23:32:49.358073 12236 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 6a3a834d77474c3d8feaa5a01f3faff3 (table default.Table_dfb60e6e40874a5da0c56f6926ce1f63 [id=1ea3dfab929845f89d4680129ae45d94]). Expected version 1 got 0
I20260810 23:32:49.359113 12236 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet ab8d987109ba40198645881b449cb972 (table default.Table_d8a0e3e655b9417d9df3ff6d82b76f15 [id=d0b86eafe9544cd98d8fce8ece3b5c10]). Expected version 1 got 0
I20260810 23:32:49.360154 14582 raft_consensus.cc:482] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:49.360737 14582 raft_consensus.cc:504] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.367146 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3a834d77474c3d8feaa5a01f3faff3" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:49.376114 14638 tablet.cc:1722] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:32:49.385075 14639 tablet.cc:1722] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:32:49.402706 14587 raft_consensus.cc:482] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:49.403326 14587 raft_consensus.cc:504] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.405789 14587 leader_election.cc:288] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:49.409303 14587 raft_consensus.cc:482] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:49.410928 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c29b11192604ac18c21b7bdc0238fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:38 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:49.409787 14587 raft_consensus.cc:504] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.412698 12769 tablet_service.cc:1514] Processing DeleteTablet for tablet 423e8d583d704fa9834026ebcd57e17e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:49.414417 14587 leader_election.cc:288] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:49.415460 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab8d987109ba40198645881b449cb972" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:49.417423 12952 leader_election.cc:341] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:49.419453 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab8d987109ba40198645881b449cb972" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:49.420231 12803 raft_consensus.cc:2454] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:49.420470 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4f55ef709d473f89a16d32160eb61b" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:49.422076 12952 leader_election.cc:341] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:49.422771 12958 leader_election.cc:302] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:49.422806 12236 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 6a3a834d77474c3d8feaa5a01f3faff3 (table default.Table_dfb60e6e40874a5da0c56f6926ce1f63 [id=1ea3dfab929845f89d4680129ae45d94]). Expected version 1 got 0
I20260810 23:32:49.423940 14587 raft_consensus.cc:2781] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:49.424317 14587 raft_consensus.cc:482] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:49.424613 14587 raft_consensus.cc:3037] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.426767 13062 raft_consensus.cc:2454] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:49.428063 13212 leader_election.cc:302] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
I20260810 23:32:49.435325 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_535019edc4634bb68e6c59f1e8c56bac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:49.435912 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4f55ef709d473f89a16d32160eb61b" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:49.436931 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet 42517f11422b42c39db282e2c5f584a3 with delete_type TABLET_DATA_DELETED (replaced by tablet a480093c481941f0bb35949c7606f37f) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:49.437583 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_535019edc4634bb68e6c59f1e8c56bac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:49.437815 12958 leader_election.cc:341] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: Tablet not RUNNING: STOPPED
I20260810 23:32:49.438341 14649 tablet.cc:1722] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:49.438266 12958 leader_election.cc:302] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:49.447399 14584 raft_consensus.cc:2726] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:49.447576 14587 raft_consensus.cc:504] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.449873 14587 leader_election.cc:288] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:49.442035 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d697e7533af4fc0a15cb41f3fcce4db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:49.437165 14645 raft_consensus.cc:2781] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:49.486888 14582 leader_election.cc:288] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:49.489728 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3a834d77474c3d8feaa5a01f3faff3" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:49.497653 14121 ts_tablet_manager.cc:1918] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:49.498200 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab8d987109ba40198645881b449cb972" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:49.501024 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_ca9eee6f19c34490b6a9cad315a8f740 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 3 got 0
I20260810 23:32:49.501269 13335 raft_consensus.cc:3037] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.502351 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
I20260810 23:32:49.503898 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 6a3a834d77474c3d8feaa5a01f3faff3 (table default.Table_dfb60e6e40874a5da0c56f6926ce1f63 [id=1ea3dfab929845f89d4680129ae45d94]). Expected version 1 got 0
I20260810 23:32:49.504199 13213 leader_election.cc:302] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:49.506150 14645 raft_consensus.cc:686] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:49.507119 14645 consensus_queue.cc:237] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.511752 13974 ts_tablet_manager.cc:1918] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:49.512241 13974 log.cc:1196] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/3d596299c82b4ff1ac151d68b97ff55b
I20260810 23:32:49.513144 13974 ts_tablet_manager.cc:1939] T 3d596299c82b4ff1ac151d68b97ff55b P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:49.516110 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 164) took 5315 ms. Trace:
W20260810 23:32:49.500114 12952 leader_election.cc:341] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:49.523032 14427 consensus_queue.cc:1035] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:49.528275 14648 tablet.cc:1722] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:49.513785 13691 tablet_bootstrap.cc:492] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:49.529884 13691 ts_tablet_manager.cc:1397] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.344s user 0.014s sys 0.002s
I20260810 23:32:49.517500 14121 log.cc:1196] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/840400119d20424ea0f03f2bff550d44
I20260810 23:32:49.533074 14121 ts_tablet_manager.cc:1939] T 840400119d20424ea0f03f2bff550d44 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
W20260810 23:32:49.538112 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 92674.1us
I20260810 23:32:49.540349 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab8d987109ba40198645881b449cb972" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:49.541164 12805 raft_consensus.cc:3037] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.544904 13691 raft_consensus.cc:348] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.545728 13691 raft_consensus.cc:374] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.505484 12234 catalog_manager.cc:5495] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b reported cstate change: term changed from 0 to 1, leader changed from <none> to f26517150f1c480a877f15aec71d020b (127.11.226.130). New cstate: current_term: 1 leader_uuid: "f26517150f1c480a877f15aec71d020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:49.546011 13691 raft_consensus.cc:729] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.559480 13514 tablet_bootstrap.cc:492] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:49.560125 12240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 240adfa4be844518bbc9624d0e408789
I20260810 23:32:49.561630 12240 catalog_manager.cc:6565] ReplaceTablet: tablet 240adfa4be844518bbc9624d0e408789 deleted and replaced by tablet 4344987a6c90428fbdf50765a4631455
I20260810 23:32:49.561995 12240 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 261) took 2037 ms. Trace:
I20260810 23:32:49.563071 12229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 08c45b71432a45269cd6a106fc5f7292
W20260810 23:32:49.564258 12958 consensus_peers.cc:487] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 133654008f5247b2ac87dcd09195f197. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:49.564666 12229 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 262) took 1963 ms. Trace:
I20260810 23:32:49.564962 12229 rpcz_store.cc:276] 0810 23:32:47.601700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.297348 (+1695648us) service_pool.cc:227] Handling call
0810 23:32:49.298746 (+ 1398us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:49.299267 (+ 521us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:49.299421 (+ 154us) catalog_manager.cc:2761] Locking tablets
0810 23:32:49.300140 (+ 719us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:49.487622 (+187482us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:49.488058 (+ 436us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:49.562958 (+ 74900us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:49.564601 (+ 1643us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:49.302660 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:49.303147 (+ 487us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:49.303175 (+ 28us) write_op.cc:435] Acquired schema lock
0810 23:32:49.303195 (+ 20us) tablet.cc:660] Decoding operations
0810 23:32:49.303580 (+ 385us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:49.303645 (+ 65us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:49.303669 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:49.303984 (+ 315us) tablet.cc:699] Row locks acquired
0810 23:32:49.304002 (+ 18us) write_op.cc:260] PREPARE: finished
0810 23:32:49.304188 (+ 186us) write_op.cc:270] Start()
0810 23:32:49.304328 (+ 140us) write_op.cc:276] Timestamp: P: 1786404769304140 usec, L: 0
0810 23:32:49.304348 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:32:49.304931 (+ 583us) log.cc:842] Serialized 774 byte log entry
0810 23:32:49.467899 (+162968us) op_driver.cc:444] REPLICATION: finished
0810 23:32:49.472638 (+ 4739us) write_op.cc:301] APPLY: starting
0810 23:32:49.472715 (+ 77us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:49.472835 (+ 120us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:49.472953 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:49.473289 (+ 336us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:49.473473 (+ 184us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:49.473522 (+ 49us) write_op.cc:312] APPLY: finished
0810 23:32:49.473767 (+ 245us) log.cc:842] Serialized 29 byte log entry
0810 23:32:49.473949 (+ 182us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.474030 (+ 81us) write_op.cc:454] Released schema lock
0810 23:32:49.474066 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:49.474317 (+ 251us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.474355 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4519,"apply.run_cpu_time_us":2154,"apply.run_wall_time_us":2152,"num_ops":2,"prepare.queue_time_us":148,"prepare.run_cpu_time_us":2567,"prepare.run_wall_time_us":2570,"raft.queue_time_us":3048,"raft.run_cpu_time_us":757,"raft.run_wall_time_us":757,"replication_time_us":163448,"spinlock_wait_cycles":1920}]]}
I20260810 23:32:49.580814 13974 rpcz_store.cc:276] 0810 23:32:44.236324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:44.249163 (+ 12839us) service_pool.cc:227] Handling call
0810 23:32:48.779242 (+4530079us) ts_tablet_manager.cc:1145] Deleting tablet 3d596299c82b4ff1ac151d68b97ff55b
0810 23:32:48.779505 (+ 263us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:49.138431 (+358926us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:49.511473 (+373042us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:49.516010 (+ 4537us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4528695}
I20260810 23:32:49.582816 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 167) took 4655 ms. Trace:
I20260810 23:32:49.583060 13008 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c45b71432a45269cd6a106fc5f7292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:49.538982 14652 consensus_queue.cc:1035] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:49.582629 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet 240adfa4be844518bbc9624d0e408789 with delete_type TABLET_DATA_DELETED (replaced by tablet 4344987a6c90428fbdf50765a4631455) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:49.585335 12226 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 263) took 1778 ms. Trace:
I20260810 23:32:49.578920 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c45b71432a45269cd6a106fc5f7292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:49.579418 12768 tablet_service.cc:1514] Processing DeleteTablet for tablet 240adfa4be844518bbc9624d0e408789 with delete_type TABLET_DATA_DELETED (replaced by tablet 4344987a6c90428fbdf50765a4631455) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:49.588380 13514 ts_tablet_manager.cc:1397] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.312s user 0.010s sys 0.003s
I20260810 23:32:49.585791 12226 rpcz_store.cc:276] 0810 23:32:47.807343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.298176 (+1490833us) service_pool.cc:227] Handling call
0810 23:32:49.325631 (+ 27455us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:49.337622 (+ 11991us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:49.584413 (+246791us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:49.584823 (+ 410us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:49.585237 (+ 414us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:49.348973 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:49.349705 (+ 732us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:49.349747 (+ 42us) write_op.cc:435] Acquired schema lock
0810 23:32:49.349772 (+ 25us) tablet.cc:660] Decoding operations
0810 23:32:49.350259 (+ 487us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:49.350321 (+ 62us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:49.350358 (+ 37us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:49.350724 (+ 366us) tablet.cc:699] Row locks acquired
0810 23:32:49.350744 (+ 20us) write_op.cc:260] PREPARE: finished
0810 23:32:49.351101 (+ 357us) write_op.cc:270] Start()
0810 23:32:49.351277 (+ 176us) write_op.cc:276] Timestamp: P: 1786404769350997 usec, L: 0
0810 23:32:49.351311 (+ 34us) op_driver.cc:331] REPLICATION: starting
0810 23:32:49.353069 (+ 1758us) log.cc:842] Serialized 433 byte log entry
0810 23:32:49.554417 (+201348us) op_driver.cc:444] REPLICATION: finished
0810 23:32:49.557615 (+ 3198us) write_op.cc:301] APPLY: starting
0810 23:32:49.557694 (+ 79us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:49.557880 (+ 186us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:49.558022 (+ 142us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:49.579471 (+ 21449us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:49.579707 (+ 236us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:49.579776 (+ 69us) write_op.cc:312] APPLY: finished
0810 23:32:49.580135 (+ 359us) log.cc:842] Serialized 29 byte log entry
0810 23:32:49.582920 (+ 2785us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.583010 (+ 90us) write_op.cc:454] Released schema lock
0810 23:32:49.584118 (+ 1108us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:49.584393 (+ 275us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.584421 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3033,"apply.run_cpu_time_us":2626,"apply.run_wall_time_us":27010,"num_ops":2,"prepare.queue_time_us":9662,"prepare.run_cpu_time_us":4452,"prepare.run_wall_time_us":4509,"raft.queue_time_us":1092,"raft.run_cpu_time_us":20,"raft.run_wall_time_us":21,"replication_time_us":203030,"threads_started":1}]]}
I20260810 23:32:49.562299 12240 rpcz_store.cc:276] 0810 23:32:47.524641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.275505 (+1750864us) service_pool.cc:227] Handling call
0810 23:32:49.561919 (+286414us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:49.305438 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:49.305890 (+ 452us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:49.305915 (+ 25us) write_op.cc:435] Acquired schema lock
0810 23:32:49.305930 (+ 15us) tablet.cc:660] Decoding operations
0810 23:32:49.307842 (+ 1912us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:49.307891 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:49.307930 (+ 39us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:49.309359 (+ 1429us) tablet.cc:699] Row locks acquired
0810 23:32:49.309380 (+ 21us) write_op.cc:260] PREPARE: finished
0810 23:32:49.309554 (+ 174us) write_op.cc:270] Start()
0810 23:32:49.309666 (+ 112us) write_op.cc:276] Timestamp: P: 1786404769309508 usec, L: 0
0810 23:32:49.309692 (+ 26us) op_driver.cc:331] REPLICATION: starting
0810 23:32:49.310342 (+ 650us) log.cc:842] Serialized 631 byte log entry
0810 23:32:49.554174 (+243832us) op_driver.cc:444] REPLICATION: finished
0810 23:32:49.555048 (+ 874us) write_op.cc:301] APPLY: starting
0810 23:32:49.555119 (+ 71us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:49.555246 (+ 127us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:49.555344 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:49.555761 (+ 417us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:49.555995 (+ 234us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:49.556067 (+ 72us) write_op.cc:312] APPLY: finished
0810 23:32:49.556366 (+ 299us) log.cc:842] Serialized 29 byte log entry
0810 23:32:49.556617 (+ 251us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.556733 (+ 116us) write_op.cc:454] Released schema lock
0810 23:32:49.556787 (+ 54us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:49.557104 (+ 317us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.557143 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":677,"apply.run_cpu_time_us":2379,"apply.run_wall_time_us":2377,"num_ops":2,"prepare.queue_time_us":710,"prepare.run_cpu_time_us":2674,"prepare.run_wall_time_us":6268,"replication_time_us":244374}]]}
W20260810 23:32:49.540604 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 112572us
I20260810 23:32:49.561566 13665 tablet_bootstrap.cc:492] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:49.596136 13665 ts_tablet_manager.cc:1397] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.500s user 0.016s sys 0.000s
I20260810 23:32:49.596141 14672 tablet.cc:1722] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:32:49.596858 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 3d596299c82b4ff1ac151d68b97ff55b (table default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]) successfully deleted
I20260810 23:32:49.605178 12805 raft_consensus.cc:2454] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:49.599479 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 145) took 6274 ms. Trace:
I20260810 23:32:49.601140 13691 consensus_queue.cc:260] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.612061 14121 rpcz_store.cc:276] 0810 23:32:43.324729 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.325228 (+ 499us) service_pool.cc:227] Handling call
0810 23:32:48.780590 (+5455362us) ts_tablet_manager.cc:1145] Deleting tablet 840400119d20424ea0f03f2bff550d44
0810 23:32:48.780642 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:49.234614 (+453972us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:49.497218 (+262604us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:49.599377 (+102159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5306876}
I20260810 23:32:49.613915 14121 tablet_replica.cc:330] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:49.614838 14121 raft_consensus.cc:2229] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:49.614938 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 840400119d20424ea0f03f2bff550d44 (table default.Table_e0603d4692934dce9e07597811ccad0a [id=ad699de357934075aee8d7152f8e130b]) successfully deleted
I20260810 23:32:49.618911 14656 raft_consensus.cc:482] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:49.619431 14656 raft_consensus.cc:504] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:49.622871 13691 ts_tablet_manager.cc:1428] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.093s user 0.004s sys 0.004s
W20260810 23:32:49.639436 12235 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6b8bf38da5647889756935404caad75 consensus state has no leader
I20260810 23:32:49.640439 12235 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 264) took 1821 ms. Trace:
I20260810 23:32:49.640794 12235 rpcz_store.cc:276] 0810 23:32:47.819252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.310400 (+1491148us) service_pool.cc:227] Handling call
0810 23:32:49.316881 (+ 6481us) spinlock_profiling.cc:243] Waited 5.43 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e77d6 00007f0ce6e2d634 00007f0ce6e2d5c2 00007f0ce6e2d4f1 00007f0ce6e2d46c 00007f0ce6e2c722 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:49.335841 (+ 18960us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:49.344039 (+ 8198us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:49.385941 (+ 41902us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:49.386170 (+ 229us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:49.638707 (+252537us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:49.640363 (+ 1656us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:49.387481 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:49.388003 (+ 522us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:49.388069 (+ 66us) write_op.cc:435] Acquired schema lock
0810 23:32:49.388081 (+ 12us) tablet.cc:660] Decoding operations
0810 23:32:49.388441 (+ 360us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:49.388488 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:49.388514 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:49.388697 (+ 183us) tablet.cc:699] Row locks acquired
0810 23:32:49.388738 (+ 41us) write_op.cc:260] PREPARE: finished
0810 23:32:49.388887 (+ 149us) write_op.cc:270] Start()
0810 23:32:49.388999 (+ 112us) write_op.cc:276] Timestamp: P: 1786404769388832 usec, L: 0
0810 23:32:49.389013 (+ 14us) op_driver.cc:331] REPLICATION: starting
0810 23:32:49.389752 (+ 739us) log.cc:842] Serialized 405 byte log entry
0810 23:32:49.635943 (+246191us) op_driver.cc:444] REPLICATION: finished
0810 23:32:49.636560 (+ 617us) write_op.cc:301] APPLY: starting
0810 23:32:49.636642 (+ 82us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:49.636771 (+ 129us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:49.636939 (+ 168us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:49.637145 (+ 206us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:49.637286 (+ 141us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:49.637329 (+ 43us) write_op.cc:312] APPLY: finished
0810 23:32:49.637538 (+ 209us) log.cc:842] Serialized 23 byte log entry
0810 23:32:49.638329 (+ 791us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.638408 (+ 79us) write_op.cc:454] Released schema lock
0810 23:32:49.638439 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":12497280,"child_traces":[["op",{"apply.queue_time_us":352,"num_ops":1,"prepare.queue_time_us":136,"prepare.run_cpu_time_us":2619,"prepare.run_wall_time_us":2619,"raft.queue_time_us":336,"raft.run_cpu_time_us":20,"raft.run_wall_time_us":20,"replication_time_us":246823}]]}
I20260810 23:32:49.646349 13276 tablet_service.cc:1514] Processing DeleteTablet for tablet 240adfa4be844518bbc9624d0e408789 with delete_type TABLET_DATA_DELETED (replaced by tablet 4344987a6c90428fbdf50765a4631455) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:49.646801 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c45b71432a45269cd6a106fc5f7292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:49.653021 13691 tablet_bootstrap.cc:492] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:49.599891 13665 raft_consensus.cc:348] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.657881 13665 raft_consensus.cc:374] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.658246 13665 raft_consensus.cc:729] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.658751 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c29b11192604ac18c21b7bdc0238fb3" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:49.660495 12803 raft_consensus.cc:2454] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:49.639544 13514 raft_consensus.cc:348] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.662463 13514 raft_consensus.cc:374] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.662796 13514 raft_consensus.cc:729] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.663579 13514 consensus_queue.cc:260] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.583331 13974 rpcz_store.cc:276] 0810 23:32:44.927541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:44.927719 (+ 178us) service_pool.cc:227] Handling call
0810 23:32:49.582367 (+4654648us) ts_tablet_manager.cc:1145] Deleting tablet 806fe5162aef4d36ad5e2ac324047426
0810 23:32:49.582429 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:49.582741 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4653920}
I20260810 23:32:49.665889 14567 raft_consensus.cc:482] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:49.669589 14567 raft_consensus.cc:504] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.670783 13514 ts_tablet_manager.cc:1428] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.082s user 0.006s sys 0.000s
I20260810 23:32:49.671795 13514 tablet_bootstrap.cc:492] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:49.672431 14567 leader_election.cc:288] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:49.667963 13935 ts_tablet_manager.cc:1918] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:49.674337 13935 log.cc:1196] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/fd4f55ef709d473f89a16d32160eb61b
I20260810 23:32:49.675334 13935 ts_tablet_manager.cc:1939] T fd4f55ef709d473f89a16d32160eb61b P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:49.677835 13514 tablet_bootstrap.cc:654] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:49.681133 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 806fe5162aef4d36ad5e2ac324047426 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 806fe5162aef4d36ad5e2ac324047426 already in progress: creating tablet
I20260810 23:32:49.683971 14638 tablet.cc:1722] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260810 23:32:49.683934 12228 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 259) took 2494 ms. Trace:
I20260810 23:32:49.616892 14121 pending_rounds.cc:62] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b: Trying to abort 1 pending ops.
I20260810 23:32:49.659173 13665 consensus_queue.cc:260] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.669219 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 168) took 4554 ms. Trace:
I20260810 23:32:49.711772 13691 tablet_bootstrap.cc:654] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.710021 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c29b11192604ac18c21b7bdc0238fb3" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:49.711964 13974 rpcz_store.cc:276] 0810 23:32:45.114495 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.114686 (+ 191us) service_pool.cc:227] Handling call
0810 23:32:49.668788 (+4554102us) ts_tablet_manager.cc:1145] Deleting tablet 4d697e7533af4fc0a15cb41f3fcce4db
0810 23:32:49.668847 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:49.669150 (+ 303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4504000}
I20260810 23:32:49.713505 13974 tablet_replica.cc:330] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:49.695288 14639 tablet.cc:1722] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260810 23:32:49.673542 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fef127263f1470ea1056a454971e465" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:49.715166 12698 leader_election.cc:341] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:49.711154 13665 ts_tablet_manager.cc:1428] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.114s user 0.006s sys 0.000s
I20260810 23:32:49.717061 13665 tablet_bootstrap.cc:492] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
W20260810 23:32:49.718751 12237 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 32) took 3336 ms (client timeout 1000 ms). Trace:
I20260810 23:32:49.722024 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 840400119d20424ea0f03f2bff550d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:49.726912 12229 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_02819bd703c44810832aecd72d795582"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:49.729044 12229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02819bd703c44810832aecd72d795582 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:49.733110 12958 leader_election.cc:302] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:49.734064 12958 leader_election.cc:302] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:49.729929 12224 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 265) took 1664 ms. Trace:
I20260810 23:32:49.735473 12224 rpcz_store.cc:276] 0810 23:32:48.065506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.345551 (+1280045us) service_pool.cc:227] Handling call
0810 23:32:49.391665 (+ 46114us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:49.393787 (+ 2122us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:49.729179 (+335392us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:49.729561 (+ 382us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:49.729862 (+ 301us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:49.395294 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:49.395927 (+ 633us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:49.395980 (+ 53us) write_op.cc:435] Acquired schema lock
0810 23:32:49.395999 (+ 19us) tablet.cc:660] Decoding operations
0810 23:32:49.396460 (+ 461us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:49.396520 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:49.396559 (+ 39us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:49.396885 (+ 326us) tablet.cc:699] Row locks acquired
0810 23:32:49.396905 (+ 20us) write_op.cc:260] PREPARE: finished
0810 23:32:49.397090 (+ 185us) write_op.cc:270] Start()
0810 23:32:49.397220 (+ 130us) write_op.cc:276] Timestamp: P: 1786404769397023 usec, L: 0
0810 23:32:49.397243 (+ 23us) op_driver.cc:331] REPLICATION: starting
0810 23:32:49.398017 (+ 774us) log.cc:842] Serialized 433 byte log entry
0810 23:32:49.722243 (+324226us) op_driver.cc:444] REPLICATION: finished
0810 23:32:49.726496 (+ 4253us) write_op.cc:301] APPLY: starting
0810 23:32:49.726570 (+ 74us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:49.726688 (+ 118us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:49.726803 (+ 115us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:49.727342 (+ 539us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:49.727521 (+ 179us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:49.727565 (+ 44us) write_op.cc:312] APPLY: finished
0810 23:32:49.727793 (+ 228us) log.cc:842] Serialized 29 byte log entry
0810 23:32:49.728610 (+ 817us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.728878 (+ 268us) write_op.cc:454] Released schema lock
0810 23:32:49.728926 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:49.730330 (+ 1404us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.730356 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4038,"apply.run_cpu_time_us":2105,"apply.run_wall_time_us":4051,"num_ops":2,"prepare.queue_time_us":114,"prepare.run_cpu_time_us":3166,"prepare.run_wall_time_us":4915,"raft.queue_time_us":132,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":13,"replication_time_us":324883,"spinlock_wait_cycles":35200}]]}
I20260810 23:32:49.739554 12224 master_service.cc:946] ReplaceTablet: received request to replace tablet 4d0d9227d8574de8bd73041f3e8043a0 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:49.747539 13514 tablet_bootstrap.cc:492] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:49.748099 13514 ts_tablet_manager.cc:1397] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.076s user 0.028s sys 0.020s
I20260810 23:32:49.732859 12240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_d8373b61098245bcaab43448a20578f6" } modify_external_catalogs: true
I20260810 23:32:49.749416 12240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_d8373b61098245bcaab43448a20578f6" } modify_external_catalogs: true
I20260810 23:32:49.751029 13514 raft_consensus.cc:348] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.751886 13514 raft_consensus.cc:374] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.752183 13665 tablet_bootstrap.cc:654] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.752271 13514 raft_consensus.cc:729] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.753106 13514 consensus_queue.cc:260] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.755662 13514 ts_tablet_manager.cc:1428] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 23:32:49.756958 13514 tablet_bootstrap.cc:492] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:49.703531 14121 pending_rounds.cc:68] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317113926601056256 op_type: NO_OP noop_request { }
I20260810 23:32:49.677908 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 166) took 4054 ms. Trace:
I20260810 23:32:49.761318 13935 rpcz_store.cc:276] 0810 23:32:45.623359 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.623530 (+ 171us) service_pool.cc:227] Handling call
0810 23:32:49.282007 (+3658477us) ts_tablet_manager.cc:1145] Deleting tablet fd4f55ef709d473f89a16d32160eb61b
0810 23:32:49.282058 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:49.573221 (+291163us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:49.667655 (+ 94434us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:49.677839 (+ 10184us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3657723}
I20260810 23:32:49.762878 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet fd4f55ef709d473f89a16d32160eb61b (table default.Table_addbc3e258944d0cb5d2e7fb11214412 [id=d4e32fc5299547268eb0c93e2d3c00e1]) successfully deleted
I20260810 23:32:49.655970 14656 leader_election.cc:288] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:49.761251 14121 raft_consensus.cc:2866] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:32:49.765826 14121 raft_consensus.cc:2258] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:49.768121 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet db363eb0f98d460eb6f5a577233d8ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:36 UTC) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:49.775401 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 4d697e7533af4fc0a15cb41f3fcce4db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d697e7533af4fc0a15cb41f3fcce4db already in progress: creating tablet
W20260810 23:32:49.778244 13212 consensus_peers.cc:487] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Couldn't send request to peer f26517150f1c480a877f15aec71d020b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 840400119d20424ea0f03f2bff550d44. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:49.779838 12952 leader_election.cc:341] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:49.719156 12237 rpcz_store.cc:269] 0810 23:32:46.365829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:46.978230 (+612401us) service_pool.cc:227] Handling call
0810 23:32:47.236455 (+258225us) spinlock_profiling.cc:243] Waited 90.8 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:49.702493 (+2466038us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":17,"spinlock_wait_cycles":208918272}
I20260810 23:32:49.793469 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 42517f11422b42c39db282e2c5f584a3 with delete_type TABLET_DATA_DELETED (replaced by tablet a480093c481941f0bb35949c7606f37f) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:49.798105 12223 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 33) took 2172 ms (client timeout 1000 ms). Trace:
W20260810 23:32:49.798637 12223 rpcz_store.cc:269] 0810 23:32:47.625559 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:47.858833 (+233274us) service_pool.cc:227] Handling call
0810 23:32:47.988127 (+129294us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:49.798007 (+1809880us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5612,"reported_tablets":19,"spinlock_wait_cycles":288786688}
I20260810 23:32:49.800454 13514 tablet_bootstrap.cc:654] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.832674 14216 raft_consensus.cc:2781] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:49.764695 14427 raft_consensus.cc:2781] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:49.844396 14427 raft_consensus.cc:482] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:49.845080 14427 raft_consensus.cc:3037] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.850812 14121 ts_tablet_manager.cc:1905] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:49.856459 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 171) took 3715 ms. Trace:
I20260810 23:32:49.856982 13935 rpcz_store.cc:276] 0810 23:32:46.141177 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:46.141579 (+ 402us) service_pool.cc:227] Handling call
0810 23:32:49.855885 (+3714306us) ts_tablet_manager.cc:1145] Deleting tablet 80b54edbb45d496c94efedb6feee8e45
0810 23:32:49.855948 (+ 63us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:49.856359 (+ 411us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3708088}
I20260810 23:32:49.858376 13935 tablet_replica.cc:330] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:49.871701 13514 tablet_bootstrap.cc:492] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:49.872344 13514 ts_tablet_manager.cc:1397] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.116s user 0.015s sys 0.010s
I20260810 23:32:49.831249 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fef127263f1470ea1056a454971e465" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:49.875209 13514 raft_consensus.cc:348] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:49.890095 13514 raft_consensus.cc:374] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.890542 13514 raft_consensus.cc:729] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.691812 12228 rpcz_store.cc:276] 0810 23:32:47.189150 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.244070 (+2054920us) service_pool.cc:227] Handling call
0810 23:32:49.245743 (+ 1673us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:49.328585 (+ 82842us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:49.385784 (+ 57199us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:49.386018 (+ 234us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:49.650947 (+264929us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:49.683842 (+ 32895us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:49.390492 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:49.390935 (+ 443us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:49.390964 (+ 29us) write_op.cc:435] Acquired schema lock
0810 23:32:49.390981 (+ 17us) tablet.cc:660] Decoding operations
0810 23:32:49.391283 (+ 302us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:49.391328 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:49.391351 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:49.391547 (+ 196us) tablet.cc:699] Row locks acquired
0810 23:32:49.391559 (+ 12us) write_op.cc:260] PREPARE: finished
0810 23:32:49.391690 (+ 131us) write_op.cc:270] Start()
0810 23:32:49.391800 (+ 110us) write_op.cc:276] Timestamp: P: 1786404769391641 usec, L: 0
0810 23:32:49.391815 (+ 15us) op_driver.cc:331] REPLICATION: starting
0810 23:32:49.393146 (+ 1331us) log.cc:842] Serialized 405 byte log entry
0810 23:32:49.636232 (+243086us) op_driver.cc:444] REPLICATION: finished
0810 23:32:49.645642 (+ 9410us) write_op.cc:301] APPLY: starting
0810 23:32:49.645707 (+ 65us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:49.645808 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:49.645930 (+ 122us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:49.646137 (+ 207us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:49.646273 (+ 136us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:49.646323 (+ 50us) write_op.cc:312] APPLY: finished
0810 23:32:49.646543 (+ 220us) log.cc:842] Serialized 23 byte log entry
0810 23:32:49.647405 (+ 862us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.647473 (+ 68us) write_op.cc:454] Released schema lock
0810 23:32:49.647533 (+ 60us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:49.647772 (+ 239us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.647797 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":76672,"child_traces":[["op",{"apply.queue_time_us":9267,"apply.run_cpu_time_us":1664,"apply.run_wall_time_us":2337,"mutex_wait_us":3028,"num_ops":1,"prepare.queue_time_us":3147,"prepare.run_cpu_time_us":2459,"prepare.run_wall_time_us":3150,"raft.queue_time_us":95,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":13,"replication_time_us":244332}]]}
W20260810 23:32:49.862998 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 80b54edbb45d496c94efedb6feee8e45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80b54edbb45d496c94efedb6feee8e45 already in progress: creating tablet
I20260810 23:32:49.864275 12236 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 266) took 1751 ms. Trace:
I20260810 23:32:49.896889 12228 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_77274019ccb14c57b2a9ce882ee84dac" } new_table_name: "default.Table_399593e632684b20a3032bb34a182dec" modify_external_catalogs: true
I20260810 23:32:49.894872 12236 rpcz_store.cc:276] 0810 23:32:48.112877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.434657 (+1321780us) service_pool.cc:227] Handling call
0810 23:32:49.438083 (+ 3426us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:49.439864 (+ 1781us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:49.863380 (+423516us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:49.863833 (+ 453us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:49.864187 (+ 354us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:49.441592 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:49.442076 (+ 484us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:49.442116 (+ 40us) write_op.cc:435] Acquired schema lock
0810 23:32:49.442140 (+ 24us) tablet.cc:660] Decoding operations
0810 23:32:49.442570 (+ 430us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:49.442629 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:49.442671 (+ 42us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:49.442921 (+ 250us) tablet.cc:699] Row locks acquired
0810 23:32:49.442946 (+ 25us) write_op.cc:260] PREPARE: finished
0810 23:32:49.443127 (+ 181us) write_op.cc:270] Start()
0810 23:32:49.443269 (+ 142us) write_op.cc:276] Timestamp: P: 1786404769443059 usec, L: 0
0810 23:32:49.443297 (+ 28us) op_driver.cc:331] REPLICATION: starting
0810 23:32:49.444102 (+ 805us) log.cc:842] Serialized 433 byte log entry
0810 23:32:49.722531 (+278429us) op_driver.cc:444] REPLICATION: finished
0810 23:32:49.730675 (+ 8144us) write_op.cc:301] APPLY: starting
0810 23:32:49.730735 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:49.730851 (+ 116us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:49.730951 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:49.803449 (+ 72498us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:49.803726 (+ 277us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:49.803799 (+ 73us) write_op.cc:312] APPLY: finished
0810 23:32:49.804280 (+ 481us) log.cc:842] Serialized 29 byte log entry
0810 23:32:49.862849 (+ 58569us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.862984 (+ 135us) write_op.cc:454] Released schema lock
0810 23:32:49.863042 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:49.863417 (+ 375us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:49.863472 (+ 55us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":8009,"apply.run_cpu_time_us":3116,"apply.run_wall_time_us":133059,"num_ops":2,"prepare.queue_time_us":103,"prepare.run_cpu_time_us":2838,"prepare.run_wall_time_us":2839,"raft.queue_time_us":206,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":12,"replication_time_us":279151}]]}
I20260810 23:32:49.873916 12237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_216b21581f454466a84891c7e3d0e567" } modify_external_catalogs: true
I20260810 23:32:49.902213 12237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_216b21581f454466a84891c7e3d0e567" } modify_external_catalogs: true
I20260810 23:32:49.914503 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_829882defbbb45f78cd9bb86c3f7303f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:49.920609 13514 consensus_queue.cc:260] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:49.941309 12700 leader_election.cc:341] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:49.941929 12700 leader_election.cc:302] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:49.943138 14678 raft_consensus.cc:2726] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:49.941891 13514 ts_tablet_manager.cc:1428] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.069s user 0.009s sys 0.000s
I20260810 23:32:49.944672 14656 raft_consensus.cc:482] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 191d120f4c7f44fc8475305a6d2e75f3)
I20260810 23:32:49.945619 14656 raft_consensus.cc:504] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:49.950305 13935 raft_consensus.cc:2229] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:49.952445 13935 raft_consensus.cc:2258] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:49.954147 13514 tablet_bootstrap.cc:492] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:49.959345 14656 leader_election.cc:288] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:49.927058 12236 master_service.cc:946] ReplaceTablet: received request to replace tablet f1e8311c941b43058384cee6d8263dc1 from {username='slave'} at 127.0.0.1:34742
W20260810 23:32:49.970293 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_829882defbbb45f78cd9bb86c3f7303f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:49.985018 13935 ts_tablet_manager.cc:1905] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:49.993891 13974 raft_consensus.cc:2229] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:49.995529 13691 tablet_bootstrap.cc:492] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:49.996170 13691 ts_tablet_manager.cc:1397] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.343s user 0.038s sys 0.040s
W20260810 23:32:50.013132 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 72257.1us
I20260810 23:32:50.010931 14663 consensus_queue.cc:1035] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260810 23:32:50.032155 14576 raft_consensus.cc:482] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:50.032979 14576 raft_consensus.cc:504] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.035986 14576 leader_election.cc:288] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:50.041569 13514 tablet_bootstrap.cc:654] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:50.044296 14666 raft_consensus.cc:482] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:50.044893 14666 raft_consensus.cc:504] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:50.047319 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b261673faa964b65af3576c89e302864" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:50.048270 13336 raft_consensus.cc:2454] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:50.050110 14666 leader_election.cc:288] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:50.051409 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d1b15dd545c4598aa5c7a9dc53618ec" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:50.056092 12698 leader_election.cc:341] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:50.059409 13213 leader_election.cc:341] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: b261673faa964b65af3576c89e302864
I20260810 23:32:50.062398 13974 raft_consensus.cc:2258] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:50.069017 14638 tablet.cc:1717] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3: Skipping requested alter to schema version 4, tablet already version 4
I20260810 23:32:50.078341 13974 ts_tablet_manager.cc:1905] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:50.087040 14639 tablet.cc:1717] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3: Skipping requested alter to schema version 4, tablet already version 4
I20260810 23:32:50.110313 12952 leader_election.cc:302] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:50.110962 13691 raft_consensus.cc:348] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.114816 14584 raft_consensus.cc:2781] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:32:50.115386 14584 raft_consensus.cc:482] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 191d120f4c7f44fc8475305a6d2e75f3)
I20260810 23:32:50.111855 13691 raft_consensus.cc:374] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:50.116361 13691 raft_consensus.cc:729] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:50.117151 13691 consensus_queue.cc:260] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.122197 13691 ts_tablet_manager.cc:1428] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.126s user 0.009s sys 0.000s
I20260810 23:32:50.123767 13691 tablet_bootstrap.cc:492] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:50.134284 14584 raft_consensus.cc:3037] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:32:50.147535 14639 tablet.cc:1722] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260810 23:32:50.148838 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 56120.1us
I20260810 23:32:50.163019 13514 tablet_bootstrap.cc:492] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:50.163585 13514 ts_tablet_manager.cc:1397] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.218s user 0.018s sys 0.012s
I20260810 23:32:50.166712 12235 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_5cde9c2beb0448c1971328536e2c47d8 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 4 got 3
I20260810 23:32:50.170984 14648 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:58598 (request call id 189) took 1048 ms. Trace:
I20260810 23:32:50.174157 13514 raft_consensus.cc:348] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.175027 13514 raft_consensus.cc:374] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:50.175302 13514 raft_consensus.cc:729] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:50.208302 13691 tablet_bootstrap.cc:654] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:50.171552 14648 rpcz_store.cc:276] 0810 23:32:49.122287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.122595 (+ 308us) service_pool.cc:227] Handling call
0810 23:32:50.170881 (+1048286us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:49.218585 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 23:32:49.219138 (+ 553us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 23:32:49.219189 (+ 51us) alter_schema_op.cc:65] Acquired schema lock
0810 23:32:49.219278 (+ 89us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 23:32:49.219458 (+ 180us) alter_schema_op.cc:126] START. Timestamp: 1786404769219346
0810 23:32:49.219513 (+ 55us) op_driver.cc:331] REPLICATION: starting
0810 23:32:49.254491 (+ 34978us) log.cc:842] Serialized 173 byte log entry
0810 23:32:49.392850 (+138359us) op_driver.cc:444] REPLICATION: finished
0810 23:32:49.499904 (+107054us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 23:32:49.530381 (+ 30477us) log.cc:842] Serialized 5 byte log entry
0810 23:32:49.931138 (+400757us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:50.170206 (+239068us) log.cc:842] Serialized 14 byte log entry
0810 23:32:50.170468 (+ 262us) alter_schema_op.cc:71] Released schema lock
0810 23:32:50.170599 (+ 131us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":106777,"prepare.queue_time_us":71857,"prepare.run_cpu_time_us":3182,"prepare.run_wall_time_us":86150,"raft.queue_time_us":9108,"raft.run_cpu_time_us":913,"raft.run_wall_time_us":926,"replication_time_us":173241,"thread_start_us":219518,"threads_started":3,"wal-append.queue_time_us":36350}]]}
I20260810 23:32:50.140872 14216 raft_consensus.cc:686] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:50.225771 14216 consensus_queue.cc:237] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.230085 12229 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 267) took 1987 ms. Trace:
I20260810 23:32:50.230520 12229 rpcz_store.cc:276] 0810 23:32:48.242241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.569081 (+1326840us) service_pool.cc:227] Handling call
0810 23:32:49.726677 (+157596us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7156 00007f0ce6e24034 00007f0ce6e23fc2 00007f0ce6e23ef1 00007f0ce6e23e6c 00007f0ce6e23122 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:49.735506 (+ 8829us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:49.737114 (+ 1608us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:50.229247 (+492133us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:50.229670 (+ 423us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:50.230000 (+ 330us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:49.740082 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:49.740592 (+ 510us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:49.740622 (+ 30us) write_op.cc:435] Acquired schema lock
0810 23:32:49.740642 (+ 20us) tablet.cc:660] Decoding operations
0810 23:32:49.741005 (+ 363us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:49.741052 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:49.741090 (+ 38us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:49.741337 (+ 247us) tablet.cc:699] Row locks acquired
0810 23:32:49.741359 (+ 22us) write_op.cc:260] PREPARE: finished
0810 23:32:49.741505 (+ 146us) write_op.cc:270] Start()
0810 23:32:49.741613 (+ 108us) write_op.cc:276] Timestamp: P: 1786404769741420 usec, L: 0
0810 23:32:49.741633 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:32:49.742298 (+ 665us) log.cc:842] Serialized 433 byte log entry
0810 23:32:50.224344 (+482046us) op_driver.cc:444] REPLICATION: finished
0810 23:32:50.224921 (+ 577us) write_op.cc:301] APPLY: starting
0810 23:32:50.225143 (+ 222us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:50.225455 (+ 312us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:50.225746 (+ 291us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:50.226673 (+ 927us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:50.227088 (+ 415us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:50.227219 (+ 131us) write_op.cc:312] APPLY: finished
0810 23:32:50.227668 (+ 449us) log.cc:842] Serialized 29 byte log entry
0810 23:32:50.228811 (+ 1143us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:50.228924 (+ 113us) write_op.cc:454] Released schema lock
0810 23:32:50.228970 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:50.229247 (+ 277us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:50.229274 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":360532864,"child_traces":[["op",{"apply.queue_time_us":173,"apply.run_cpu_time_us":3810,"apply.run_wall_time_us":4588,"num_ops":2,"prepare.queue_time_us":1632,"prepare.run_cpu_time_us":2478,"prepare.run_wall_time_us":2478,"replication_time_us":482562}]]}
I20260810 23:32:50.235321 12224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 4d0d9227d8574de8bd73041f3e8043a0
I20260810 23:32:50.237123 12224 catalog_manager.cc:6565] ReplaceTablet: tablet 4d0d9227d8574de8bd73041f3e8043a0 deleted and replaced by tablet fb8d482c6ee54e45a41201d98a5b60b4
I20260810 23:32:50.237829 12757 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d0d9227d8574de8bd73041f3e8043a0 with delete_type TABLET_DATA_DELETED (replaced by tablet fb8d482c6ee54e45a41201d98a5b60b4) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:50.241173 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d0d9227d8574de8bd73041f3e8043a0 with delete_type TABLET_DATA_DELETED (replaced by tablet fb8d482c6ee54e45a41201d98a5b60b4) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:50.233886 14688 raft_consensus.cc:482] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:50.250010 14664 raft_consensus.cc:2726] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:50.247514 14688 raft_consensus.cc:504] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.257527 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_cb78723192e84474b32e919ec39a002b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:50.260124 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb78723192e84474b32e919ec39a002b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:50.256382 13665 tablet_bootstrap.cc:492] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:50.262876 12229 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_9e92a0778ad2454eb9ffe997c50710c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:50.264817 13665 ts_tablet_manager.cc:1397] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.548s user 0.016s sys 0.006s
W20260810 23:32:50.265952 12229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e92a0778ad2454eb9ffe997c50710c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:50.277611 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d0d9227d8574de8bd73041f3e8043a0 with delete_type TABLET_DATA_DELETED (replaced by tablet fb8d482c6ee54e45a41201d98a5b60b4) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:50.279031 13213 leader_election.cc:341] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: fd4f55ef709d473f89a16d32160eb61b
I20260810 23:32:50.285032 12235 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_4612ad4a086f41b28be40a81f4b0a41c" } new_table_name: "default.Table_eb8f4b9d6a8d4c9891bff8243880751d" modify_external_catalogs: true
I20260810 23:32:50.287734 14648 tablet.cc:1722] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260810 23:32:50.289242 13691 tablet_bootstrap.cc:492] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:50.289803 13691 ts_tablet_manager.cc:1397] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.166s user 0.014s sys 0.012s
I20260810 23:32:50.293131 12240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet f1e8311c941b43058384cee6d8263dc1
I20260810 23:32:50.293676 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ad5d63a2104995990a86a000963cdc with delete_type TABLET_DATA_DELETED (replaced by tablet 8fb2ee1c9b41478da001aa67d7a7ad3c) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:50.296123 12240 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 268) took 1189 ms. Trace:
I20260810 23:32:50.297287 12757 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e8311c941b43058384cee6d8263dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:50.300230 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e8311c941b43058384cee6d8263dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:50.296475 12240 rpcz_store.cc:276] 0810 23:32:49.106273 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.591909 (+485636us) service_pool.cc:227] Handling call
0810 23:32:49.732738 (+140829us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7466 00007f0ce6e28b34 00007f0ce6e28ac2 00007f0ce6e289f1 00007f0ce6e2896c 00007f0ce6e27c22 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:49.750209 (+ 17471us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:49.750821 (+ 612us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:49.751004 (+ 183us) catalog_manager.cc:2761] Locking tablets
0810 23:32:49.751783 (+ 779us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:50.291587 (+539804us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:50.292073 (+ 486us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:50.293021 (+ 948us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:50.296043 (+ 3022us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:49.779608 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:49.780732 (+ 1124us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:49.780785 (+ 53us) write_op.cc:435] Acquired schema lock
0810 23:32:49.780834 (+ 49us) tablet.cc:660] Decoding operations
0810 23:32:49.781630 (+ 796us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:49.781699 (+ 69us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:49.781800 (+ 101us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:49.782643 (+ 843us) tablet.cc:699] Row locks acquired
0810 23:32:49.782665 (+ 22us) write_op.cc:260] PREPARE: finished
0810 23:32:49.783003 (+ 338us) write_op.cc:270] Start()
0810 23:32:49.783169 (+ 166us) write_op.cc:276] Timestamp: P: 1786404769782866 usec, L: 0
0810 23:32:49.783221 (+ 52us) op_driver.cc:331] REPLICATION: starting
0810 23:32:49.863963 (+ 80742us) log.cc:842] Serialized 734 byte log entry
0810 23:32:50.256449 (+392486us) op_driver.cc:444] REPLICATION: finished
0810 23:32:50.277662 (+ 21213us) write_op.cc:301] APPLY: starting
0810 23:32:50.277740 (+ 78us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:50.278005 (+ 265us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:50.278168 (+ 163us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:50.278626 (+ 458us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:50.278923 (+ 297us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:50.279009 (+ 86us) write_op.cc:312] APPLY: finished
0810 23:32:50.279346 (+ 337us) log.cc:842] Serialized 29 byte log entry
0810 23:32:50.288454 (+ 9108us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:50.288585 (+ 131us) write_op.cc:454] Released schema lock
0810 23:32:50.288650 (+ 65us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":321234304,"child_traces":[["op",{"apply.queue_time_us":20976,"num_ops":2,"prepare.queue_time_us":167,"prepare.run_cpu_time_us":5504,"prepare.run_wall_time_us":86741,"replication_time_us":473127}]]}
I20260810 23:32:50.273247 14688 leader_election.cc:288] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:50.348389 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 133654008f5247b2ac87dcd09195f197 with delete_type TABLET_DATA_DELETED (replaced by tablet ca8c0d1dacc344cbacb0c5988f0e2221) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:50.350557 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f1452c292e9c476b9e26f56998f0ad55 consensus state has no leader
I20260810 23:32:50.360296 14576 raft_consensus.cc:482] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:50.361202 14576 raft_consensus.cc:504] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:50.362226 12228 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dfda619f19a5459494b1e4f6d22821cc consensus state has no leader
W20260810 23:32:50.365634 12234 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 34) took 1678 ms (client timeout 1000 ms). Trace:
I20260810 23:32:50.372305 13514 consensus_queue.cc:260] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.384014 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e8311c941b43058384cee6d8263dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:50.384188 13665 raft_consensus.cc:348] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.385371 13665 raft_consensus.cc:374] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:50.385733 13665 raft_consensus.cc:729] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:50.386546 13665 consensus_queue.cc:260] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.388348 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d4c879ca6a402dbafe91940e7b367a" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:50.389778 13665 ts_tablet_manager.cc:1428] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.125s user 0.010s sys 0.000s
I20260810 23:32:50.390436 14576 leader_election.cc:288] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:32:50.392647 12698 leader_election.cc:341] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:50.404932 13665 tablet_bootstrap.cc:492] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:50.406200 13514 ts_tablet_manager.cc:1428] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.242s user 0.008s sys 0.004s
I20260810 23:32:50.430339 13514 tablet_bootstrap.cc:492] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:32:50.409793 12234 rpcz_store.cc:269] 0810 23:32:48.687131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.207719 (+520588us) service_pool.cc:227] Handling call
0810 23:32:50.365530 (+1157811us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:49.572381 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:49.572871 (+ 490us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:49.572901 (+ 30us) write_op.cc:435] Acquired schema lock
0810 23:32:49.572915 (+ 14us) tablet.cc:660] Decoding operations
0810 23:32:49.573260 (+ 345us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:49.573310 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:49.573337 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:49.573544 (+ 207us) tablet.cc:699] Row locks acquired
0810 23:32:49.573555 (+ 11us) write_op.cc:260] PREPARE: finished
0810 23:32:49.573711 (+ 156us) write_op.cc:270] Start()
0810 23:32:49.573819 (+ 108us) write_op.cc:276] Timestamp: P: 1786404769573630 usec, L: 0
0810 23:32:49.573835 (+ 16us) op_driver.cc:331] REPLICATION: starting
0810 23:32:49.723393 (+149558us) log.cc:842] Serialized 536 byte log entry
0810 23:32:49.724252 (+ 859us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:50.149813 (+425561us) op_driver.cc:444] REPLICATION: finished
0810 23:32:50.150582 (+ 769us) write_op.cc:301] APPLY: starting
0810 23:32:50.150663 (+ 81us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:50.150856 (+ 193us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:50.150909 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:50.151218 (+ 309us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:50.151472 (+ 254us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:50.151572 (+ 100us) write_op.cc:312] APPLY: finished
0810 23:32:50.151962 (+ 390us) log.cc:842] Serialized 23 byte log entry
0810 23:32:50.152353 (+ 391us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:50.152449 (+ 96us) write_op.cc:454] Released schema lock
0810 23:32:50.152517 (+ 68us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:50.152958 (+ 441us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:50.153015 (+ 57us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":155,"apply.run_cpu_time_us":2688,"apply.run_wall_time_us":2688,"num_ops":1,"prepare.queue_time_us":282,"prepare.run_cpu_time_us":5373,"prepare.run_wall_time_us":152038,"replication_time_us":575868,"spinlock_wait_cycles":342308096}]]}
I20260810 23:32:50.440811 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_b89242e803f549b8aca7b725f00cd775"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:50.449249 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b89242e803f549b8aca7b725f00cd775 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:50.336905 14427 raft_consensus.cc:504] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:50.459612 13691 raft_consensus.cc:348] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.460749 13691 raft_consensus.cc:374] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:50.461089 13691 raft_consensus.cc:729] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:50.462568 13691 consensus_queue.cc:260] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.354722 14656 consensus_queue.cc:1035] T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:50.469167 14678 raft_consensus.cc:482] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:50.470062 14678 raft_consensus.cc:504] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.474908 14678 leader_election.cc:288] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:50.475538 14686 raft_consensus.cc:482] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:50.479122 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5794637d5dd43f2b05edd53865f4e99" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:50.481174 12698 leader_election.cc:341] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:50.314709 14584 raft_consensus.cc:504] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:50.476352 14686 raft_consensus.cc:504] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.494445 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b261673faa964b65af3576c89e302864" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:50.495472 13060 raft_consensus.cc:2379] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in current term 2: Already voted for candidate f26517150f1c480a877f15aec71d020b in this term.
I20260810 23:32:50.488631 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d1b15dd545c4598aa5c7a9dc53618ec" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:50.513365 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "423e8d583d704fa9834026ebcd57e17e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:50.524711 12952 leader_election.cc:341] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:50.528414 14686 leader_election.cc:288] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:50.533618 14584 leader_election.cc:288] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:50.535148 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b261673faa964b65af3576c89e302864" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:50.537009 13335 raft_consensus.cc:3037] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:32:50.549685 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4f55ef709d473f89a16d32160eb61b" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:50.551116 13054 raft_consensus.cc:2454] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
W20260810 23:32:50.558347 12235 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45e39a7b58ac402da7c5e845ea6c8329 consensus state has no leader
I20260810 23:32:50.558565 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c29b11192604ac18c21b7bdc0238fb3" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:50.560237 14427 leader_election.cc:288] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:50.562219 13212 leader_election.cc:302] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
W20260810 23:32:50.560874 12952 leader_election.cc:341] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:50.562880 13691 ts_tablet_manager.cc:1428] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.273s user 0.009s sys 0.007s
I20260810 23:32:50.565062 13691 tablet_bootstrap.cc:492] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
W20260810 23:32:50.567544 12700 leader_election.cc:341] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:50.568140 12700 leader_election.cc:302] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:50.569070 14678 raft_consensus.cc:2726] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:50.582334 13935 ts_tablet_manager.cc:1918] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:50.582849 13935 log.cc:1196] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/d30bc9bd4e1f462b8d3be7f8a2f28f1e
I20260810 23:32:50.593542 13935 ts_tablet_manager.cc:1939] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:50.600790 13514 tablet_bootstrap.cc:654] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:50.602957 13691 tablet_bootstrap.cc:654] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:50.599375 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 180) took 3654 ms. Trace:
I20260810 23:32:50.603869 13935 rpcz_store.cc:276] 0810 23:32:46.944941 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:46.945691 (+ 750us) service_pool.cc:227] Handling call
0810 23:32:49.858168 (+2912477us) ts_tablet_manager.cc:1145] Deleting tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e
0810 23:32:49.858226 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:50.317555 (+459329us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:50.582025 (+264470us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:50.599286 (+ 17261us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39831,"tablet-delete.queue_time_us":2911598}
I20260810 23:32:50.606004 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e (table default.Table_10156b698d054ae8845a67255a9daf3d [id=c48d5d1f759b42f3a5c5a6c64588b30f]) successfully deleted
I20260810 23:32:50.609114 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "423e8d583d704fa9834026ebcd57e17e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:50.609977 12802 raft_consensus.cc:2454] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
W20260810 23:32:50.612605 12218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.188:45477, user_credentials={real_user=slave}} blocked reactor thread for 119784us
I20260810 23:32:50.619139 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 181) took 3499 ms. Trace:
I20260810 23:32:50.620172 12229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_32d7e9316d054deca361c6e3d9b5e1f4" } modify_external_catalogs: true
I20260810 23:32:50.620800 12229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_32d7e9316d054deca361c6e3d9b5e1f4" } modify_external_catalogs: true
I20260810 23:32:50.624954 12235 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_c5dbbcbf87314046b7357e16f8b94e35" } new_table_name: "default.Table_cd6a61f95ef74a6495c279e07bfd8b35" modify_external_catalogs: true
I20260810 23:32:50.623977 14121 ts_tablet_manager.cc:1918] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:50.664999 14121 log.cc:1196] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/440de7d9de8d4be9a5322e1fe3388b9b
I20260810 23:32:50.666163 14121 ts_tablet_manager.cc:1939] T 440de7d9de8d4be9a5322e1fe3388b9b P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
W20260810 23:32:50.666850 12235 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f4c2810b36f4e7a93443bdc1a07485a consensus state has no leader
W20260810 23:32:50.642889 12958 consensus_peers.cc:487] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:50.673208 12958 leader_election.cc:341] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: b261673faa964b65af3576c89e302864
I20260810 23:32:50.675347 12958 leader_election.cc:302] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:50.619535 13935 rpcz_store.cc:276] 0810 23:32:47.119606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:47.119800 (+ 194us) service_pool.cc:227] Handling call
0810 23:32:50.618223 (+3498423us) ts_tablet_manager.cc:1145] Deleting tablet 4cdec13ee70d402fa3a08119e7bf1313
0810 23:32:50.618712 (+ 489us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:50.619065 (+ 353us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3484813}
W20260810 23:32:50.677237 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 4cdec13ee70d402fa3a08119e7bf1313 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cdec13ee70d402fa3a08119e7bf1313 already in progress: creating tablet
I20260810 23:32:50.689913 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 182) took 3316 ms. Trace:
I20260810 23:32:50.690516 13935 rpcz_store.cc:276] 0810 23:32:47.373591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:47.374336 (+ 745us) service_pool.cc:227] Handling call
0810 23:32:50.683220 (+3308884us) ts_tablet_manager.cc:1145] Deleting tablet f47b8b57d5b44a018bc2d7b3f976d803
0810 23:32:50.683320 (+ 100us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:50.689796 (+ 6476us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3301044}
W20260810 23:32:50.707480 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet f47b8b57d5b44a018bc2d7b3f976d803 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f47b8b57d5b44a018bc2d7b3f976d803 already in progress: creating tablet
I20260810 23:32:50.615563 13665 tablet_bootstrap.cc:654] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:50.622445 13213 consensus_peers.cc:487] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42517f11422b42c39db282e2c5f584a3. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:50.716404 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 94189.3us
I20260810 23:32:50.717494 12229 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 4344987a6c90428fbdf50765a4631455
I20260810 23:32:50.624735 13974 ts_tablet_manager.cc:1918] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:50.638836 14688 raft_consensus.cc:482] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:50.732127 14688 raft_consensus.cc:504] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.735203 14688 leader_election.cc:288] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
W20260810 23:32:50.738657 13213 leader_election.cc:341] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 440de7d9de8d4be9a5322e1fe3388b9b
I20260810 23:32:50.739476 13974 log.cc:1196] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/840400119d20424ea0f03f2bff550d44
I20260810 23:32:50.740578 13974 ts_tablet_manager.cc:1939] T 840400119d20424ea0f03f2bff550d44 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:50.744597 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea (DEFAULT_TABLE table=default.Table_7751e2b5b7db4c62beba2c5564a6affa [id=575025603ae44b0785de76cc5a28ab9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:50.745882 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:50.707726 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 194) took 1851 ms. Trace:
I20260810 23:32:50.751667 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea (DEFAULT_TABLE table=default.Table_7751e2b5b7db4c62beba2c5564a6affa [id=575025603ae44b0785de76cc5a28ab9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:50.751797 13935 rpcz_store.cc:276] 0810 23:32:48.856365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:48.874497 (+ 18132us) service_pool.cc:227] Handling call
0810 23:32:50.706909 (+1832412us) ts_tablet_manager.cc:1145] Deleting tablet 62705f99687340aaa1aca0942b28487f
0810 23:32:50.706980 (+ 71us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:50.707628 (+ 648us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1830264}
I20260810 23:32:50.752713 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 4f453bd72168429086837b2b9b09fc78 (DEFAULT_TABLE table=default.Table_2e4541bbe3aa45c1a1c85533c2ed79f3 [id=c9327e173e574f75b4886311e0933acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:50.753185 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:50.753247 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 202) took 1587 ms. Trace:
W20260810 23:32:50.753414 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 62705f99687340aaa1aca0942b28487f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62705f99687340aaa1aca0942b28487f already in progress: creating tablet
I20260810 23:32:50.753806 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f453bd72168429086837b2b9b09fc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:50.753618 13935 rpcz_store.cc:276] 0810 23:32:49.166321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.166497 (+ 176us) service_pool.cc:227] Handling call
0810 23:32:50.752916 (+1586419us) ts_tablet_manager.cc:1145] Deleting tablet 124259164adf404c9cbe5f4e0d5e16c0
0810 23:32:50.752962 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:50.753185 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1585634}
W20260810 23:32:50.754930 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 124259164adf404c9cbe5f4e0d5e16c0 because the tablet was not found. No further retry: Not found: Tablet not found: 124259164adf404c9cbe5f4e0d5e16c0
I20260810 23:32:50.758157 13514 tablet_bootstrap.cc:492] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:50.758646 13514 ts_tablet_manager.cc:1397] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.352s user 0.012s sys 0.011s
I20260810 23:32:50.767123 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 169) took 5645 ms. Trace:
I20260810 23:32:50.767071 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea (DEFAULT_TABLE table=default.Table_7751e2b5b7db4c62beba2c5564a6affa [id=575025603ae44b0785de76cc5a28ab9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:50.768337 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:50.767717 13974 rpcz_store.cc:276] 0810 23:32:45.121481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.121650 (+ 169us) service_pool.cc:227] Handling call
0810 23:32:49.713288 (+4591638us) ts_tablet_manager.cc:1145] Deleting tablet 840400119d20424ea0f03f2bff550d44
0810 23:32:49.713345 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:50.158429 (+445084us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:50.624443 (+466014us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:50.767028 (+142585us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63733,"tablet-delete.queue_time_us":4548207}
I20260810 23:32:50.772678 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 170) took 5650 ms. Trace:
I20260810 23:32:50.773056 13974 rpcz_store.cc:276] 0810 23:32:45.122508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.122687 (+ 179us) service_pool.cc:227] Handling call
0810 23:32:50.772291 (+5649604us) ts_tablet_manager.cc:1145] Deleting tablet 81e26fe4d42445279f12bb17b2841d08
0810 23:32:50.772375 (+ 84us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:50.772606 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5606997}
W20260810 23:32:50.774508 14706 log.cc:924] Time spent T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.088s user 0.002s sys 0.000s
I20260810 23:32:50.778131 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 146) took 7430 ms. Trace:
I20260810 23:32:50.778544 14121 rpcz_store.cc:276] 0810 23:32:43.347486 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.347654 (+ 168us) service_pool.cc:227] Handling call
0810 23:32:49.613667 (+6266013us) ts_tablet_manager.cc:1145] Deleting tablet 440de7d9de8d4be9a5322e1fe3388b9b
0810 23:32:49.613765 (+ 98us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:50.106911 (+493146us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:50.623448 (+516537us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:50.778044 (+154596us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8635,"tablet-delete.queue_time_us":6139717}
I20260810 23:32:50.773154 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 840400119d20424ea0f03f2bff550d44 (table default.Table_e0603d4692934dce9e07597811ccad0a [id=ad699de357934075aee8d7152f8e130b]) successfully deleted
W20260810 23:32:50.780632 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 81e26fe4d42445279f12bb17b2841d08 because the tablet was not found. No further retry: Not found: Tablet not found: 81e26fe4d42445279f12bb17b2841d08
I20260810 23:32:50.781349 12237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 0be35dabbec44caebd1df687b167f34a
I20260810 23:32:50.782945 12237 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 270) took 1010 ms. Trace:
W20260810 23:32:50.784595 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 109340us
I20260810 23:32:50.785089 14216 raft_consensus.cc:2781] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:50.785431 14216 raft_consensus.cc:482] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:50.785742 14216 raft_consensus.cc:3037] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:50.785811 12958 leader_election.cc:341] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: b261673faa964b65af3576c89e302864
I20260810 23:32:50.783284 12237 rpcz_store.cc:276] 0810 23:32:49.772430 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.787851 (+ 15421us) service_pool.cc:227] Handling call
0810 23:32:49.873777 (+ 85926us) spinlock_profiling.cc:243] Waited 84.3 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7466 00007f0ce6e28b34 00007f0ce6e28ac2 00007f0ce6e289f1 00007f0ce6e2896c 00007f0ce6e27c22 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:49.902752 (+ 28975us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:49.903476 (+ 724us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:49.903617 (+ 141us) catalog_manager.cc:2761] Locking tablets
0810 23:32:50.723373 (+819756us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:50.780828 (+ 57455us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:50.781097 (+ 269us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:50.781243 (+ 146us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:50.782875 (+ 1632us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:50.731565 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:50.732326 (+ 761us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:50.732373 (+ 47us) write_op.cc:435] Acquired schema lock
0810 23:32:50.732401 (+ 28us) tablet.cc:660] Decoding operations
0810 23:32:50.733033 (+ 632us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:50.733115 (+ 82us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:50.733159 (+ 44us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:50.733700 (+ 541us) tablet.cc:699] Row locks acquired
0810 23:32:50.733729 (+ 29us) write_op.cc:260] PREPARE: finished
0810 23:32:50.733939 (+ 210us) write_op.cc:270] Start()
0810 23:32:50.734097 (+ 158us) write_op.cc:276] Timestamp: P: 1786404770733856 usec, L: 0
0810 23:32:50.734128 (+ 31us) op_driver.cc:331] REPLICATION: starting
0810 23:32:50.735112 (+ 984us) log.cc:842] Serialized 734 byte log entry
0810 23:32:50.776982 (+ 41870us) op_driver.cc:444] REPLICATION: finished
0810 23:32:50.777306 (+ 324us) write_op.cc:301] APPLY: starting
0810 23:32:50.777376 (+ 70us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:50.777513 (+ 137us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:50.777623 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:50.777903 (+ 280us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:50.778070 (+ 167us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:50.778116 (+ 46us) write_op.cc:312] APPLY: finished
0810 23:32:50.778378 (+ 262us) log.cc:842] Serialized 29 byte log entry
0810 23:32:50.778544 (+ 166us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:50.778624 (+ 80us) write_op.cc:454] Released schema lock
0810 23:32:50.778664 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:50.778874 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:50.778897 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":193780480,"child_traces":[["op",{"apply.queue_time_us":110,"apply.run_cpu_time_us":1818,"apply.run_wall_time_us":1818,"num_ops":2,"prepare.queue_time_us":5092,"raft.queue_time_us":90,"raft.run_cpu_time_us":870,"raft.run_wall_time_us":876,"replication_time_us":42744}]]}
I20260810 23:32:50.788071 12768 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be35dabbec44caebd1df687b167f34a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:50.792279 13276 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be35dabbec44caebd1df687b167f34a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:50.796252 13665 tablet_bootstrap.cc:492] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:50.799130 14648 tablet.cc:1717] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b: Skipping requested alter to schema version 4, tablet already version 4
I20260810 23:32:50.800518 13665 ts_tablet_manager.cc:1397] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.396s user 0.011s sys 0.003s
I20260810 23:32:50.805058 13007 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be35dabbec44caebd1df687b167f34a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:50.809806 13335 raft_consensus.cc:2454] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 2.
I20260810 23:32:50.810328 14664 raft_consensus.cc:2726] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:32:50.812455 13665 raft_consensus.cc:348] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.813190 13665 raft_consensus.cc:374] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:50.813442 13665 raft_consensus.cc:729] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:50.814057 13665 consensus_queue.cc:260] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.825297 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d4c879ca6a402dbafe91940e7b367a" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:50.827244 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 440de7d9de8d4be9a5322e1fe3388b9b (table default.Table_3b251f8589be43f29704b8906d49f6cd [id=c56022cc8cad4c7cbefafad0782cefde]) successfully deleted
I20260810 23:32:50.828759 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 147) took 7446 ms. Trace:
I20260810 23:32:50.829267 14121 rpcz_store.cc:276] 0810 23:32:43.382228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.382423 (+ 195us) service_pool.cc:227] Handling call
0810 23:32:50.828226 (+7445803us) ts_tablet_manager.cc:1145] Deleting tablet 4f3049bcdc6e43c795961c74f0d008f5
0810 23:32:50.828299 (+ 73us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:50.828687 (+ 388us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7351326}
W20260810 23:32:50.830879 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 4f3049bcdc6e43c795961c74f0d008f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f3049bcdc6e43c795961c74f0d008f5 already in progress: creating tablet
I20260810 23:32:50.832110 14121 tablet_replica.cc:330] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:50.832728 13665 ts_tablet_manager.cc:1428] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260810 23:32:50.832798 13691 tablet_bootstrap.cc:492] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:50.838425 13691 ts_tablet_manager.cc:1397] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.274s user 0.022s sys 0.008s
I20260810 23:32:50.839576 13974 tablet_replica.cc:330] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:50.840878 13974 raft_consensus.cc:2229] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:50.841527 13974 raft_consensus.cc:2258] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:50.847839 12240 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_5cde9c2beb0448c1971328536e2c47d8 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 4 got 3
I20260810 23:32:50.851049 13665 tablet_bootstrap.cc:492] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:50.855710 13212 leader_election.cc:302] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
W20260810 23:32:50.856438 13212 raft_consensus.cc:2670] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260810 23:32:50.857438 14121 raft_consensus.cc:2229] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:50.859081 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c29b11192604ac18c21b7bdc0238fb3" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:50.859954 12805 raft_consensus.cc:3037] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:50.866649 12700 leader_election.cc:341] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:50.867240 12700 leader_election.cc:302] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:50.868897 14576 raft_consensus.cc:2726] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:50.882831 12805 raft_consensus.cc:2454] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:50.914271 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_68bcc5970e834ef8bb810ac4f663c1d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:50.917183 12229 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_dfb60e6e40874a5da0c56f6926ce1f63" } new_table_name: "default.Table_7221fbab5f0b4fa892342870790a2f7e" modify_external_catalogs: true
I20260810 23:32:50.917764 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_5a0a5857ffc649eb96a5e9e271b8ce7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:50.918963 12226 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_10156b698d054ae8845a67255a9daf3d" } new_table_name: "default.Table_730f4da35e68448585507866c203a391" modify_external_catalogs: true
W20260810 23:32:50.919400 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a0a5857ffc649eb96a5e9e271b8ce7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:50.921034 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_4f9225f25cc842fbad1e9ce8c1dba105"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:50.923170 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f9225f25cc842fbad1e9ce8c1dba105 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:50.923975 14121 raft_consensus.cc:2258] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:50.924921 12234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_dc5d83ea7f1441fa9e2b2536bab49724" } modify_external_catalogs: true
I20260810 23:32:50.925367 12240 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
W20260810 23:32:50.925458 12217 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 67341.6us
I20260810 23:32:50.925467 12234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_dc5d83ea7f1441fa9e2b2536bab49724" } modify_external_catalogs: true
W20260810 23:32:50.927678 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68bcc5970e834ef8bb810ac4f663c1d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:50.928419 12240 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet ab8d987109ba40198645881b449cb972 (table default.Table_d8a0e3e655b9417d9df3ff6d82b76f15 [id=d0b86eafe9544cd98d8fce8ece3b5c10]). Expected version 1 got 0
I20260810 23:32:50.869995 12952 leader_election.cc:302] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3
W20260810 23:32:50.934255 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 64445.7us
I20260810 23:32:50.947134 14584 raft_consensus.cc:2781] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:32:50.949611 14584 raft_consensus.cc:686] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 2 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:50.954087 14584 consensus_queue.cc:237] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:50.962059 13974 ts_tablet_manager.cc:1905] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:50.916429 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_98f0ace30b644009a7f4cd8cbe85217c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:50.975592 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98f0ace30b644009a7f4cd8cbe85217c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:50.985704 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 4f453bd72168429086837b2b9b09fc78 (DEFAULT_TABLE table=default.Table_2e4541bbe3aa45c1a1c85533c2ed79f3 [id=c9327e173e574f75b4886311e0933acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:50.987986 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f453bd72168429086837b2b9b09fc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:50.876607 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5794637d5dd43f2b05edd53865f4e99" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:51.006038 14686 raft_consensus.cc:482] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:51.007402 14686 raft_consensus.cc:504] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.010742 14121 ts_tablet_manager.cc:1905] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:51.012514 14686 leader_election.cc:288] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:51.016849 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd9d90e1ceab443dbec12b2c81b975e8" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:51.019639 12952 leader_election.cc:341] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:50.972153 14722 log.cc:924] Time spent T 4b71d565669f40d8a6d06ab267069430 P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.096s user 0.002s sys 0.000s
I20260810 23:32:50.986502 13007 tablet_service.cc:1467] Processing CreateTablet for tablet 4f453bd72168429086837b2b9b09fc78 (DEFAULT_TABLE table=default.Table_2e4541bbe3aa45c1a1c85533c2ed79f3 [id=c9327e173e574f75b4886311e0933acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:51.024885 13007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f453bd72168429086837b2b9b09fc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:51.029330 14678 raft_consensus.cc:482] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:51.029945 14678 raft_consensus.cc:504] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.034253 14678 leader_election.cc:288] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:51.034612 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b8bf38da5647889756935404caad75" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:51.037528 12698 leader_election.cc:341] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:51.038153 14216 raft_consensus.cc:504] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:50.994975 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 200) took 1829 ms. Trace:
I20260810 23:32:51.041059 13935 rpcz_store.cc:276] 0810 23:32:49.165659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.172140 (+ 6481us) service_pool.cc:227] Handling call
0810 23:32:50.754449 (+1582309us) ts_tablet_manager.cc:1145] Deleting tablet 874ca0c3586b41a8a3074d8988813633
0810 23:32:50.754499 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:50.994865 (+240366us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1581665}
I20260810 23:32:51.036566 13514 raft_consensus.cc:348] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.063714 13665 tablet_bootstrap.cc:654] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:51.069013 13514 raft_consensus.cc:374] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:51.055917 14664 consensus_queue.cc:1035] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260810 23:32:51.074348 14216 leader_election.cc:288] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:51.075265 13514 raft_consensus.cc:729] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:51.077725 13935 tablet_replica.cc:330] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
W20260810 23:32:51.078050 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 874ca0c3586b41a8a3074d8988813633 because the tablet was not found. No further retry: Not found: Tablet not found: 874ca0c3586b41a8a3074d8988813633
I20260810 23:32:51.078678 13935 raft_consensus.cc:2229] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:51.076388 13514 consensus_queue.cc:260] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.079242 13935 raft_consensus.cc:2258] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:51.080741 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "423e8d583d704fa9834026ebcd57e17e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:51.093227 13514 ts_tablet_manager.cc:1428] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.332s user 0.008s sys 0.000s
W20260810 23:32:51.094468 12700 leader_election.cc:341] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:51.094964 12700 leader_election.cc:302] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:51.096650 14678 raft_consensus.cc:2726] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:51.049180 13691 raft_consensus.cc:348] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.100886 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1f45f0c2714a74b1fce8aaac4cc558 (DEFAULT_TABLE table=default.Table_c5751d9b450d4231a2d59983159349b7 [id=27425440b77a4601964b0eea5ba2fee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:51.102267 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1f45f0c2714a74b1fce8aaac4cc558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:51.105104 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd9d90e1ceab443dbec12b2c81b975e8" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:51.106153 12805 raft_consensus.cc:2454] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
W20260810 23:32:51.102527 12952 leader_election.cc:341] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:51.111819 13691 raft_consensus.cc:374] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 23:32:51.112669 13212 leader_election.cc:341] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Not found: Tablet not found: 440de7d9de8d4be9a5322e1fe3388b9b
I20260810 23:32:51.113138 13212 leader_election.cc:302] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:51.114656 14664 raft_consensus.cc:2726] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:51.121220 13514 tablet_bootstrap.cc:492] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:51.121879 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1f45f0c2714a74b1fce8aaac4cc558 (DEFAULT_TABLE table=default.Table_c5751d9b450d4231a2d59983159349b7 [id=27425440b77a4601964b0eea5ba2fee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:51.123371 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1f45f0c2714a74b1fce8aaac4cc558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:51.124807 13691 raft_consensus.cc:729] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:51.136863 13974 ts_tablet_manager.cc:1918] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:51.136446 13691 consensus_queue.cc:260] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.147604 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b8bf38da5647889756935404caad75" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:51.148907 13062 raft_consensus.cc:2454] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:32:51.149729 12236 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_dfb60e6e40874a5da0c56f6926ce1f63 [id=1ea3dfab929845f89d4680129ae45d94] alter complete (version 1)
I20260810 23:32:51.154522 13974 log.cc:1196] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/fd4f55ef709d473f89a16d32160eb61b
I20260810 23:32:51.155604 13974 ts_tablet_manager.cc:1939] T fd4f55ef709d473f89a16d32160eb61b P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:51.156584 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1f45f0c2714a74b1fce8aaac4cc558 (DEFAULT_TABLE table=default.Table_c5751d9b450d4231a2d59983159349b7 [id=27425440b77a4601964b0eea5ba2fee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:51.157842 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1f45f0c2714a74b1fce8aaac4cc558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:51.173558 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 173) took 5552 ms. Trace:
I20260810 23:32:51.174192 14121 ts_tablet_manager.cc:1918] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:51.174368 13514 tablet_bootstrap.cc:654] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:51.174783 14121 log.cc:1196] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/874ca0c3586b41a8a3074d8988813633
I20260810 23:32:51.174973 12768 tablet_service.cc:1467] Processing CreateTablet for tablet d603569c54ef4ffba9941d0cc3e0f97f (DEFAULT_TABLE table=default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:51.175683 14121 ts_tablet_manager.cc:1939] T 874ca0c3586b41a8a3074d8988813633 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:51.176596 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d603569c54ef4ffba9941d0cc3e0f97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:51.176604 13974 rpcz_store.cc:276] 0810 23:32:45.621432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.621674 (+ 242us) service_pool.cc:227] Handling call
0810 23:32:50.839373 (+5217699us) ts_tablet_manager.cc:1145] Deleting tablet fd4f55ef709d473f89a16d32160eb61b
0810 23:32:50.839429 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:51.066775 (+227346us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:51.136574 (+ 69799us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:51.173422 (+ 36848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5215914}
I20260810 23:32:51.177961 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet fd4f55ef709d473f89a16d32160eb61b (table default.Table_addbc3e258944d0cb5d2e7fb11214412 [id=d4e32fc5299547268eb0c93e2d3c00e1]) successfully deleted
I20260810 23:32:51.180007 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 179) took 5021 ms. Trace:
I20260810 23:32:51.180370 13974 rpcz_store.cc:276] 0810 23:32:46.159059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:46.165386 (+ 6327us) service_pool.cc:227] Handling call
0810 23:32:51.179571 (+5014185us) ts_tablet_manager.cc:1145] Deleting tablet 80b54edbb45d496c94efedb6feee8e45
0810 23:32:51.179623 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:51.179936 (+ 313us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5013458}
I20260810 23:32:51.181424 13974 tablet_replica.cc:330] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:51.182265 13974 raft_consensus.cc:2229] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:51.182875 13974 raft_consensus.cc:2258] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:51.175894 13665 tablet_bootstrap.cc:492] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:51.183735 13665 ts_tablet_manager.cc:1397] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.333s user 0.023s sys 0.007s
I20260810 23:32:51.185001 13974 ts_tablet_manager.cc:1905] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:51.189563 13665 raft_consensus.cc:348] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:51.190302 13665 raft_consensus.cc:374] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:51.190548 13665 raft_consensus.cc:729] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:51.191180 13665 consensus_queue.cc:260] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:51.205492 13665 ts_tablet_manager.cc:1428] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.021s user 0.003s sys 0.002s
I20260810 23:32:51.206542 13665 tablet_bootstrap.cc:492] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:51.209880 13514 tablet_bootstrap.cc:492] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:51.210381 13514 ts_tablet_manager.cc:1397] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.089s user 0.012s sys 0.002s
I20260810 23:32:51.213203 13514 raft_consensus.cc:348] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.213958 13514 raft_consensus.cc:374] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:51.214217 13514 raft_consensus.cc:729] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:51.214717 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 152) took 7425 ms. Trace:
I20260810 23:32:51.223335 13665 tablet_bootstrap.cc:654] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:51.227705 13514 consensus_queue.cc:260] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.238117 13935 ts_tablet_manager.cc:1905] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:51.238792 13665 tablet_bootstrap.cc:492] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:51.239266 13665 ts_tablet_manager.cc:1397] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.004s
I20260810 23:32:51.241978 13665 raft_consensus.cc:348] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.242671 13665 raft_consensus.cc:374] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:51.242899 13665 raft_consensus.cc:729] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:51.215157 14121 rpcz_store.cc:276] 0810 23:32:43.789555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.800886 (+ 11331us) service_pool.cc:227] Handling call
0810 23:32:50.831881 (+7030995us) ts_tablet_manager.cc:1145] Deleting tablet 874ca0c3586b41a8a3074d8988813633
0810 23:32:50.831948 (+ 67us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:51.092010 (+260062us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:51.143734 (+ 51724us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:51.214617 (+ 70883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6992709}
I20260810 23:32:51.243542 13665 consensus_queue.cc:260] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.254304 13665 ts_tablet_manager.cc:1428] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260810 23:32:51.255434 13665 tablet_bootstrap.cc:492] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
W20260810 23:32:51.258332 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 50116.4us
I20260810 23:32:51.255324 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 153) took 6238 ms. Trace:
I20260810 23:32:51.260155 14121 rpcz_store.cc:276] 0810 23:32:45.016506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.016708 (+ 202us) service_pool.cc:227] Handling call
0810 23:32:51.252095 (+6235387us) ts_tablet_manager.cc:1145] Deleting tablet 806fe5162aef4d36ad5e2ac324047426
0810 23:32:51.254824 (+ 2729us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:51.255239 (+ 415us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6199802}
I20260810 23:32:51.268462 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 155) took 6062 ms. Trace:
I20260810 23:32:51.269584 13665 tablet_bootstrap.cc:654] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:51.271023 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 874ca0c3586b41a8a3074d8988813633 (table default.Table_b57f0f6e94384d22af5d3bf7817d5044 [id=4bb9a6fbe6e644c583a468af86950cef]) successfully deleted
W20260810 23:32:51.272118 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 806fe5162aef4d36ad5e2ac324047426 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 806fe5162aef4d36ad5e2ac324047426 already in progress: creating tablet
I20260810 23:32:51.288831 13514 ts_tablet_manager.cc:1428] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.078s user 0.009s sys 0.000s
I20260810 23:32:51.296589 13514 tablet_bootstrap.cc:492] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:51.314460 12224 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_5cde9c2beb0448c1971328536e2c47d8 [id=b5cc65e503f7413fbade0c94e2e74b14] alter complete (version 4)
I20260810 23:32:51.322214 12764 tablet_service.cc:1467] Processing CreateTablet for tablet d97b1c97910e450b8bc42e0c7a970ce0 (DEFAULT_TABLE table=default.Table_10156b698d054ae8845a67255a9daf3d [id=c48d5d1f759b42f3a5c5a6c64588b30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:51.323701 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97b1c97910e450b8bc42e0c7a970ce0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:51.368392 12236 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 29) took 1036 ms (client timeout 1000 ms). Trace:
W20260810 23:32:51.368860 12236 rpcz_store.cc:269] 0810 23:32:50.331609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.332664 (+ 1055us) service_pool.cc:227] Handling call
0810 23:32:50.358442 (+ 25778us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:51.368303 (+1009861us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:50.912792 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:50.989042 (+ 76250us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:50.989095 (+ 53us) write_op.cc:435] Acquired schema lock
0810 23:32:50.989119 (+ 24us) tablet.cc:660] Decoding operations
0810 23:32:51.043703 (+ 54584us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:51.043809 (+ 106us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:51.043873 (+ 64us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:51.073643 (+ 29770us) tablet.cc:699] Row locks acquired
0810 23:32:51.073701 (+ 58us) write_op.cc:260] PREPARE: finished
0810 23:32:51.074017 (+ 316us) write_op.cc:270] Start()
0810 23:32:51.074189 (+ 172us) write_op.cc:276] Timestamp: P: 1786404771073915 usec, L: 0
0810 23:32:51.074213 (+ 24us) op_driver.cc:331] REPLICATION: starting
0810 23:32:51.075502 (+ 1289us) log.cc:842] Serialized 356 byte log entry
0810 23:32:51.145571 (+ 70069us) op_driver.cc:444] REPLICATION: finished
0810 23:32:51.146051 (+ 480us) write_op.cc:301] APPLY: starting
0810 23:32:51.146139 (+ 88us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:51.146357 (+ 218us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:51.146510 (+ 153us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:51.146886 (+ 376us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:51.147155 (+ 269us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:51.147260 (+ 105us) write_op.cc:312] APPLY: finished
0810 23:32:51.147633 (+ 373us) log.cc:842] Serialized 23 byte log entry
0810 23:32:51.147898 (+ 265us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:51.148001 (+ 103us) write_op.cc:454] Released schema lock
0810 23:32:51.148076 (+ 75us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:51.148425 (+ 349us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:51.148468 (+ 43us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"spinlock_wait_cycles":51379968,"child_traces":[["op",{"apply.queue_time_us":209,"apply.run_cpu_time_us":2656,"apply.run_wall_time_us":2656,"num_ops":1,"prepare.queue_time_us":169129,"prepare.run_cpu_time_us":5152,"prepare.run_wall_time_us":220691,"raft.queue_time_us":216,"raft.run_cpu_time_us":1186,"raft.run_wall_time_us":1187,"replication_time_us":71233}]]}
I20260810 23:32:51.378340 12769 tablet_service.cc:1467] Processing CreateTablet for tablet e6a4d55a0b94451688cdf293ff505ed8 (DEFAULT_TABLE table=default.Table_c4f98a2612f6454fb7ca19de9ba29ab3 [id=092fca6f55f1456cb053678e1bb500cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:51.379592 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a4d55a0b94451688cdf293ff505ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:51.276759 13031 tablet_service.cc:1467] Processing CreateTablet for tablet d603569c54ef4ffba9941d0cc3e0f97f (DEFAULT_TABLE table=default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:51.288283 13691 ts_tablet_manager.cc:1428] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.450s user 0.012s sys 0.000s
W20260810 23:32:51.388656 13691 ts_tablet_manager.cc:1460] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b: Tablet startup took 549ms
I20260810 23:32:51.268712 12958 leader_election.cc:302] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:51.325338 13008 tablet_service.cc:1467] Processing CreateTablet for tablet d97b1c97910e450b8bc42e0c7a970ce0 (DEFAULT_TABLE table=default.Table_10156b698d054ae8845a67255a9daf3d [id=c48d5d1f759b42f3a5c5a6c64588b30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:51.388837 14121 rpcz_store.cc:276] 0810 23:32:45.205725 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.207240 (+ 1515us) service_pool.cc:227] Handling call
0810 23:32:51.267880 (+6060640us) ts_tablet_manager.cc:1145] Deleting tablet 8d2f8688fc6046d58009800c5e288302
0810 23:32:51.267955 (+ 75us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:51.268367 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6046643}
W20260810 23:32:51.389035 13691 ts_tablet_manager.cc:1462] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b: Trace:
0810 23:32:37.993122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:37.993297 (+ 175us) service_pool.cc:227] Handling call
0810 23:32:37.995178 (+ 1881us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:37.995280 (+ 102us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:38.099804 (+104524us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:38.302829 (+203025us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:38.303635 (+ 806us) inbound_call.cc:173] Queueing success response
0810 23:32:50.564571 (+12260936us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0810 23:32:50.838751 (+274180us) ts_tablet_manager.cc:1429] Starting tablet replica
0810 23:32:50.891443 (+ 52692us) tablet_replica.cc:259] Starting instrumentation
0810 23:32:50.978707 (+ 87264us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":208,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":353,"tablet-open.queue_time_us":12261083}
W20260810 23:32:51.390796 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 8d2f8688fc6046d58009800c5e288302 because the tablet was not found. No further retry: Not found: Tablet not found: 8d2f8688fc6046d58009800c5e288302
W20260810 23:32:51.391619 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 123008us
I20260810 23:32:51.392249 14584 raft_consensus.cc:2781] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:51.393128 13514 tablet_bootstrap.cc:654] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:51.398164 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8fb2ee1c9b41478da001aa67d7a7ad3c consensus state has no leader
W20260810 23:32:51.408712 12226 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d97b1c97910e450b8bc42e0c7a970ce0 consensus state has no leader
I20260810 23:32:51.415200 12958 leader_election.cc:302] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:51.417753 14584 raft_consensus.cc:686] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:51.417640 13023 tablet_service.cc:1467] Processing CreateTablet for tablet e6a4d55a0b94451688cdf293ff505ed8 (DEFAULT_TABLE table=default.Table_c4f98a2612f6454fb7ca19de9ba29ab3 [id=092fca6f55f1456cb053678e1bb500cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:51.417775 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d603569c54ef4ffba9941d0cc3e0f97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:51.418717 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a4d55a0b94451688cdf293ff505ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:51.418744 14584 consensus_queue.cc:237] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:51.419839 13691 tablet_bootstrap.cc:492] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:51.422894 14686 raft_consensus.cc:2781] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:51.423380 14686 raft_consensus.cc:482] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:51.423791 14686 raft_consensus.cc:3037] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:51.424338 13008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97b1c97910e450b8bc42e0c7a970ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:51.431932 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "423e8d583d704fa9834026ebcd57e17e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:51.432736 12805 raft_consensus.cc:3037] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:51.441955 13514 tablet_bootstrap.cc:492] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:51.442493 13514 ts_tablet_manager.cc:1397] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.147s user 0.012s sys 0.000s
I20260810 23:32:51.446062 13691 tablet_bootstrap.cc:654] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:51.453662 12805 raft_consensus.cc:2454] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:51.454991 12700 leader_election.cc:302] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:51.457190 13514 raft_consensus.cc:348] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.475476 13514 raft_consensus.cc:374] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:51.475773 13514 raft_consensus.cc:729] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:51.476504 13514 consensus_queue.cc:260] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.477221 14678 raft_consensus.cc:2781] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:51.477785 14678 raft_consensus.cc:482] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:51.478268 14678 raft_consensus.cc:3037] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:51.480728 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 156) took 6272 ms. Trace:
I20260810 23:32:51.481161 14121 rpcz_store.cc:276] 0810 23:32:45.208315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.208955 (+ 640us) service_pool.cc:227] Handling call
0810 23:32:51.480176 (+6271221us) ts_tablet_manager.cc:1145] Deleting tablet fd110664bac34f52bfe922a3a76c6290
0810 23:32:51.480259 (+ 83us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:51.480649 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6170447}
I20260810 23:32:51.482908 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 157) took 6273 ms. Trace:
W20260810 23:32:51.490267 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet fd110664bac34f52bfe922a3a76c6290 because the tablet was not found. No further retry: Not found: Tablet not found: fd110664bac34f52bfe922a3a76c6290
I20260810 23:32:51.493789 14427 raft_consensus.cc:482] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:51.494625 14427 raft_consensus.cc:504] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.474516 13935 ts_tablet_manager.cc:1918] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:51.496379 13935 log.cc:1196] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/13ad5d63a2104995990a86a000963cdc
I20260810 23:32:51.497649 13935 ts_tablet_manager.cc:1939] T 13ad5d63a2104995990a86a000963cdc P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:51.497453 12226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_d942bf134f9849718bb9eb12c24806a1" } modify_external_catalogs: true
I20260810 23:32:51.498090 12226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_d942bf134f9849718bb9eb12c24806a1" } modify_external_catalogs: true
I20260810 23:32:51.498605 14427 leader_election.cc:288] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:51.483330 14121 rpcz_store.cc:276] 0810 23:32:45.209655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.210375 (+ 720us) service_pool.cc:227] Handling call
0810 23:32:51.482276 (+6271901us) ts_tablet_manager.cc:1145] Deleting tablet c45f9dfd03764d7fb1b6fb5e28b66bd3
0810 23:32:51.482430 (+ 154us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:51.482835 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6240644}
I20260810 23:32:51.505379 14121 tablet_replica.cc:330] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b: stopping tablet replica
W20260810 23:32:51.505973 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet c45f9dfd03764d7fb1b6fb5e28b66bd3 because the tablet was not found. No further retry: Not found: Tablet not found: c45f9dfd03764d7fb1b6fb5e28b66bd3
I20260810 23:32:51.506266 14121 raft_consensus.cc:2229] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:51.520411 14121 pending_rounds.cc:62] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b: Trying to abort 1 pending ops.
I20260810 23:32:51.520725 14121 pending_rounds.cc:68] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317113930038153216 op_type: NO_OP noop_request { }
I20260810 23:32:51.521170 14121 raft_consensus.cc:2866] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:32:51.521593 14121 raft_consensus.cc:2258] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:51.542158 14121 ts_tablet_manager.cc:1905] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:51.550227 12236 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:51.561908 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 201) took 2395 ms. Trace:
W20260810 23:32:51.562614 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 86008.5us
I20260810 23:32:51.564273 13514 ts_tablet_manager.cc:1428] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.121s user 0.016s sys 0.004s
I20260810 23:32:51.562356 13935 rpcz_store.cc:276] 0810 23:32:49.165990 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.172906 (+ 6916us) service_pool.cc:227] Handling call
0810 23:32:51.077462 (+1904556us) ts_tablet_manager.cc:1145] Deleting tablet 13ad5d63a2104995990a86a000963cdc
0810 23:32:51.077548 (+ 86us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:51.420919 (+343371us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:51.474109 (+ 53190us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:51.561793 (+ 87684us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1903851}
I20260810 23:32:51.566391 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 13ad5d63a2104995990a86a000963cdc (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5]) successfully deleted
I20260810 23:32:51.568346 14686 raft_consensus.cc:504] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.569901 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0dd19112504f8ca57f64dcf0bd50a8" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:51.570711 12802 raft_consensus.cc:2454] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:51.581478 13514 tablet_bootstrap.cc:492] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:51.586381 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 204) took 2409 ms. Trace:
I20260810 23:32:51.586820 13935 rpcz_store.cc:276] 0810 23:32:49.177354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.177498 (+ 144us) service_pool.cc:227] Handling call
0810 23:32:51.565881 (+2388383us) ts_tablet_manager.cc:1145] Deleting tablet 440de7d9de8d4be9a5322e1fe3388b9b
0810 23:32:51.586003 (+ 20122us) spinlock_profiling.cc:243] Waited 6.81 ms on lock 0x7b5800001898. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cd72acf11 00007f3cd7430851 00007f3cd7430e82 00007f3ce02f08b9 00007f3cdf3245d8 00007f3cdf3e8389 00007f3cdf3e8983 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49
0810 23:32:51.586068 (+ 65us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:51.586300 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15656192,"tablet-delete.queue_time_us":2387602}
W20260810 23:32:51.589123 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 440de7d9de8d4be9a5322e1fe3388b9b because the tablet was not found. No further retry: Not found: Tablet not found: 440de7d9de8d4be9a5322e1fe3388b9b
I20260810 23:32:51.596740 13514 tablet_bootstrap.cc:654] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:51.602346 13935 tablet_replica.cc:330] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:51.603399 13935 raft_consensus.cc:2229] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:51.604534 13935 raft_consensus.cc:2258] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:51.604805 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 80b54edbb45d496c94efedb6feee8e45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80b54edbb45d496c94efedb6feee8e45 already in progress: creating tablet
I20260810 23:32:51.590462 14427 consensus_queue.cc:1035] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:51.606860 13935 ts_tablet_manager.cc:1905] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:51.595335 13691 tablet_bootstrap.cc:492] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:51.610375 13691 ts_tablet_manager.cc:1397] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.191s user 0.009s sys 0.015s
I20260810 23:32:51.616595 13691 raft_consensus.cc:348] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.619517 13276 tablet_service.cc:1467] Processing CreateTablet for tablet e6a4d55a0b94451688cdf293ff505ed8 (DEFAULT_TABLE table=default.Table_c4f98a2612f6454fb7ca19de9ba29ab3 [id=092fca6f55f1456cb053678e1bb500cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:51.622252 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a4d55a0b94451688cdf293ff505ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:51.628728 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4f55ef709d473f89a16d32160eb61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:51.629413 13283 tablet_service.cc:1467] Processing CreateTablet for tablet d603569c54ef4ffba9941d0cc3e0f97f (DEFAULT_TABLE table=default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:51.629081 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd9d90e1ceab443dbec12b2c81b975e8" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:51.630894 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d603569c54ef4ffba9941d0cc3e0f97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:51.630756 13282 tablet_service.cc:1467] Processing CreateTablet for tablet d97b1c97910e450b8bc42e0c7a970ce0 (DEFAULT_TABLE table=default.Table_10156b698d054ae8845a67255a9daf3d [id=c48d5d1f759b42f3a5c5a6c64588b30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:51.615856 14678 raft_consensus.cc:504] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.631990 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97b1c97910e450b8bc42e0c7a970ce0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:51.649698 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 86099.8us
I20260810 23:32:51.651172 14678 leader_election.cc:288] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:51.658680 14664 raft_consensus.cc:482] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:51.662797 13514 tablet_bootstrap.cc:492] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:51.663420 13514 ts_tablet_manager.cc:1397] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.004s
I20260810 23:32:51.666285 13514 raft_consensus.cc:348] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.680228 13514 raft_consensus.cc:374] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:51.680790 13514 raft_consensus.cc:729] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:51.681685 13514 consensus_queue.cc:260] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.691166 12235 master_service.cc:946] ReplaceTablet: received request to replace tablet 4d1d3bd06a9a4627a1a7e83b22dba2af from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:51.617867 14686 leader_election.cc:288] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
W20260810 23:32:51.647069 12952 leader_election.cc:341] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:51.693461 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 46471.2us
I20260810 23:32:51.603950 12958 leader_election.cc:302] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:32:51.695618 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 91778.1us
I20260810 23:32:51.695785 14427 raft_consensus.cc:2781] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:51.696362 14427 raft_consensus.cc:686] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:51.697336 14427 consensus_queue.cc:237] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.696367 12958 leader_election.cc:302] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:51.712203 13514 ts_tablet_manager.cc:1428] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.048s user 0.002s sys 0.005s
I20260810 23:32:51.714803 13514 tablet_bootstrap.cc:492] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:51.721520 13691 raft_consensus.cc:374] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:51.721890 13691 raft_consensus.cc:729] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:51.723868 14686 raft_consensus.cc:2781] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:51.724380 14686 raft_consensus.cc:482] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:51.724984 14686 raft_consensus.cc:3037] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:51.733763 13691 consensus_queue.cc:260] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.735190 12223 master_service.cc:946] ReplaceTablet: received request to replace tablet a480093c481941f0bb35949c7606f37f from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:51.739262 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd9d90e1ceab443dbec12b2c81b975e8" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:51.740007 12802 raft_consensus.cc:3037] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:51.741478 14678 raft_consensus.cc:482] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:51.741971 14678 raft_consensus.cc:504] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.744369 14678 leader_election.cc:288] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:51.632055 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0dd19112504f8ca57f64dcf0bd50a8" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:51.642319 13335 raft_consensus.cc:3037] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:51.748879 14666 raft_consensus.cc:482] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:51.749482 14666 raft_consensus.cc:504] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.752998 14666 leader_election.cc:288] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:51.659579 14664 raft_consensus.cc:504] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.757277 14678 raft_consensus.cc:482] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:51.757943 14678 raft_consensus.cc:504] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.762204 14678 leader_election.cc:288] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:51.762624 12802 raft_consensus.cc:2454] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:51.769461 12237 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_74329e5e7599420e83c0c8646d466e5b" } new_table_name: "default.Table_767939c31e6e4afcaf3d29cef1e53985" modify_external_catalogs: true
I20260810 23:32:51.781399 14664 leader_election.cc:288] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:51.842067 13665 tablet_bootstrap.cc:492] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:51.842816 13665 ts_tablet_manager.cc:1397] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.588s user 0.019s sys 0.007s
I20260810 23:32:51.838104 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db363eb0f98d460eb6f5a577233d8ac0" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:51.838445 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b71d565669f40d8a6d06ab267069430" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:51.844264 13514 tablet_bootstrap.cc:654] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:51.699940 12228 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]). Expected version 1 got 0
I20260810 23:32:51.865398 14664 raft_consensus.cc:482] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:51.866029 14664 raft_consensus.cc:504] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:51.869604 14664 leader_election.cc:288] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
W20260810 23:32:51.746642 12952 consensus_peers.cc:487] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:51.873747 12226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 1b1f24e226b94ccf8ad5a656c80a5b3b
W20260810 23:32:51.874168 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 127712us
W20260810 23:32:51.875211 12952 consensus_peers.cc:487] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:51.875926 12952 leader_election.cc:341] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:51.854347 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b71d565669f40d8a6d06ab267069430" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:51.854853 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db363eb0f98d460eb6f5a577233d8ac0" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:51.856215 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b8bf38da5647889756935404caad75" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:51.879753 13062 raft_consensus.cc:3037] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:51.880888 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:51 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:51.882511 12235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 4d1d3bd06a9a4627a1a7e83b22dba2af
I20260810 23:32:51.884895 12235 catalog_manager.cc:6565] ReplaceTablet: tablet 4d1d3bd06a9a4627a1a7e83b22dba2af deleted and replaced by tablet 6596f813c59943d4a78bd813cab7aa8d
I20260810 23:32:51.885031 12769 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1d3bd06a9a4627a1a7e83b22dba2af with delete_type TABLET_DATA_DELETED (replaced by tablet 6596f813c59943d4a78bd813cab7aa8d) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:51.888540 13062 raft_consensus.cc:2454] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:32:51.889721 13691 ts_tablet_manager.cc:1428] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.276s user 0.003s sys 0.005s
I20260810 23:32:51.895015 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1d3bd06a9a4627a1a7e83b22dba2af with delete_type TABLET_DATA_DELETED (replaced by tablet 6596f813c59943d4a78bd813cab7aa8d) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:51.899906 14686 raft_consensus.cc:504] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.904721 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b8bf38da5647889756935404caad75" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:51.908164 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10033af79e854103bc47a046d2219ca0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:51.909276 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:51 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:51.910758 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b81e7056a441d2854ba5ab6b4c5433" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:51.923213 13935 ts_tablet_manager.cc:1918] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:51.923938 13935 log.cc:1196] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/db363eb0f98d460eb6f5a577233d8ac0
I20260810 23:32:51.924994 13935 ts_tablet_manager.cc:1939] T db363eb0f98d460eb6f5a577233d8ac0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
W20260810 23:32:51.930509 12698 leader_election.cc:341] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:51.938264 12700 leader_election.cc:302] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:51.939616 14678 raft_consensus.cc:2781] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:51.940228 14678 raft_consensus.cc:686] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Becoming Leader. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Running, Role: LEADER
I20260810 23:32:51.941370 14678 consensus_queue.cc:237] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:51.943960 13514 tablet_bootstrap.cc:492] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:51.944558 13514 ts_tablet_manager.cc:1397] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.230s user 0.007s sys 0.009s
I20260810 23:32:51.954813 12223 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet a480093c481941f0bb35949c7606f37f
I20260810 23:32:51.956436 12223 catalog_manager.cc:6565] ReplaceTablet: tablet a480093c481941f0bb35949c7606f37f deleted and replaced by tablet 0703cf71490c4c71a95f55863583a03c
I20260810 23:32:51.955940 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_ac935ba0ebf44788b4f41765bc914fe9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:51.955978 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_fc0d875790454fcba23aa9c7e2ee1c4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:51.957865 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac935ba0ebf44788b4f41765bc914fe9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:51.957886 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc0d875790454fcba23aa9c7e2ee1c4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:51.962037 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet a480093c481941f0bb35949c7606f37f with delete_type TABLET_DATA_DELETED (replaced by tablet 0703cf71490c4c71a95f55863583a03c) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:51.963207 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet a480093c481941f0bb35949c7606f37f with delete_type TABLET_DATA_DELETED (replaced by tablet 0703cf71490c4c71a95f55863583a03c) from {username='slave'} at 127.0.0.1:47546
W20260810 23:32:51.993896 12237 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet caee6f3203e049568c4c7cd25684c015 consensus state has no leader
W20260810 23:32:51.997001 12698 leader_election.cc:341] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:51.853469 12228 catalog_manager.cc:5495] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b reported cstate change: term changed from 0 to 1, leader changed from <none> to f26517150f1c480a877f15aec71d020b (127.11.226.130). New cstate: current_term: 1 leader_uuid: "f26517150f1c480a877f15aec71d020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } }
W20260810 23:32:51.998222 12698 leader_election.cc:341] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:51.952555 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b81e7056a441d2854ba5ab6b4c5433" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:51.999111 12228 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet ab8d987109ba40198645881b449cb972 (table default.Table_d8a0e3e655b9417d9df3ff6d82b76f15 [id=d0b86eafe9544cd98d8fce8ece3b5c10]). Expected version 1 got 0
I20260810 23:32:51.953373 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10033af79e854103bc47a046d2219ca0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:52.006067 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 203) took 2751 ms. Trace:
I20260810 23:32:52.006747 13935 rpcz_store.cc:276] 0810 23:32:49.177030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.178364 (+ 1334us) service_pool.cc:227] Handling call
0810 23:32:51.587921 (+2409557us) ts_tablet_manager.cc:1145] Deleting tablet db363eb0f98d460eb6f5a577233d8ac0
0810 23:32:51.587965 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:51.709561 (+121596us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:51.922797 (+213236us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:51.927953 (+ 5156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2408885}
I20260810 23:32:52.008891 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet db363eb0f98d460eb6f5a577233d8ac0 (table default.Table_e81793ce40c14cfe81a319f4cbd088bf [id=cca554c88a7844d5aa3312b12e89369e]) successfully deleted
I20260810 23:32:52.009609 13691 tablet_bootstrap.cc:492] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:52.010581 13514 raft_consensus.cc:348] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.011673 13514 raft_consensus.cc:374] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:52.023741 13514 raft_consensus.cc:729] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:52.024755 13514 consensus_queue.cc:260] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.026487 13665 raft_consensus.cc:348] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:52.023274 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 206) took 2824 ms. Trace:
I20260810 23:32:52.035681 12958 leader_election.cc:302] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:52.037084 12228 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:52.032356 13935 rpcz_store.cc:276] 0810 23:32:49.198603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.198765 (+ 162us) service_pool.cc:227] Handling call
0810 23:32:52.022816 (+2824051us) ts_tablet_manager.cc:1145] Deleting tablet 133654008f5247b2ac87dcd09195f197
0810 23:32:52.022908 (+ 92us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.023181 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2814465}
I20260810 23:32:52.037923 14216 raft_consensus.cc:2781] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:52.038297 12234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet fd9d90e1ceab443dbec12b2c81b975e8
W20260810 23:32:52.038496 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 133654008f5247b2ac87dcd09195f197 because the tablet was not found. No further retry: Not found: Tablet not found: 133654008f5247b2ac87dcd09195f197
I20260810 23:32:52.038767 14216 raft_consensus.cc:686] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:52.040614 12229 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 281) took 1220 ms. Trace:
W20260810 23:32:52.032068 14786 log.cc:924] Time spent T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
I20260810 23:32:52.050203 14216 consensus_queue.cc:237] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.063369 12234 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 286) took 1139 ms. Trace:
I20260810 23:32:52.063769 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_a8675270899846aaa4209e810eb7614b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:52.064366 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_c244137f10db487f95395a64261b74b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:52.065357 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet fd9d90e1ceab443dbec12b2c81b975e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='slave'} at 127.0.0.1:47546
W20260810 23:32:52.065902 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8675270899846aaa4209e810eb7614b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:52.063788 12234 rpcz_store.cc:276] 0810 23:32:50.923962 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.924148 (+ 186us) service_pool.cc:227] Handling call
0810 23:32:50.926119 (+ 1971us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:50.926669 (+ 550us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:50.926814 (+ 145us) catalog_manager.cc:2761] Locking tablets
0810 23:32:50.927353 (+ 539us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:51.507902 (+580549us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:51.534878 (+ 26976us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:52.038174 (+503296us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:52.063273 (+ 25099us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:51.363978 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:51.364746 (+ 768us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:51.364780 (+ 34us) write_op.cc:435] Acquired schema lock
0810 23:32:51.364799 (+ 19us) tablet.cc:660] Decoding operations
0810 23:32:51.365588 (+ 789us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:51.365643 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:51.365669 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:51.366036 (+ 367us) tablet.cc:699] Row locks acquired
0810 23:32:51.366054 (+ 18us) write_op.cc:260] PREPARE: finished
0810 23:32:51.366223 (+ 169us) write_op.cc:270] Start()
0810 23:32:51.366334 (+ 111us) write_op.cc:276] Timestamp: P: 1786404771366125 usec, L: 0
0810 23:32:51.366352 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 23:32:51.367070 (+ 718us) log.cc:842] Serialized 765 byte log entry
0810 23:32:51.505016 (+137946us) op_driver.cc:444] REPLICATION: finished
0810 23:32:51.505281 (+ 265us) write_op.cc:301] APPLY: starting
0810 23:32:51.505346 (+ 65us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:51.505480 (+ 134us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:51.505590 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:51.505883 (+ 293us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:51.506041 (+ 158us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:51.506090 (+ 49us) write_op.cc:312] APPLY: finished
0810 23:32:51.506313 (+ 223us) log.cc:842] Serialized 29 byte log entry
0810 23:32:51.506481 (+ 168us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:51.506558 (+ 77us) write_op.cc:454] Released schema lock
0810 23:32:51.506595 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:51.506823 (+ 228us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:51.506845 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":90,"apply.run_cpu_time_us":1772,"apply.run_wall_time_us":1766,"num_ops":2,"prepare.queue_time_us":434418,"prepare.run_cpu_time_us":3390,"prepare.run_wall_time_us":4894,"raft.queue_time_us":22723,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":7,"replication_time_us":138567}]]}
I20260810 23:32:52.048894 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet fd9d90e1ceab443dbec12b2c81b975e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:52.073276 13665 raft_consensus.cc:374] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:52.073820 13665 raft_consensus.cc:729] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:52.074656 13665 consensus_queue.cc:260] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:52.081051 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0dd19112504f8ca57f64dcf0bd50a8" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
W20260810 23:32:52.082755 12224 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 56) took 1889 ms (client timeout 1000 ms). Trace:
I20260810 23:32:52.052582 14686 leader_election.cc:288] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:52.084991 12769 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d4c879ca6a402dbafe91940e7b367a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:39 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:52.087059 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ad5d63a2104995990a86a000963cdc with delete_type TABLET_DATA_DELETED (replaced by tablet 8fb2ee1c9b41478da001aa67d7a7ad3c) from {username='slave'} at 127.0.0.1:47546
W20260810 23:32:52.060786 13213 leader_election.cc:341] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: Tablet not RUNNING: STOPPED
I20260810 23:32:52.062249 13335 raft_consensus.cc:3037] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:52.090610 12236 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 57) took 863 ms (client timeout 1000 ms). Trace:
W20260810 23:32:52.091014 12236 rpcz_store.cc:269] 0810 23:32:51.227169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:51.375328 (+148159us) service_pool.cc:227] Handling call
0810 23:32:52.090499 (+715171us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":9}
I20260810 23:32:52.041105 12229 rpcz_store.cc:276] 0810 23:32:50.820545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.821129 (+ 584us) service_pool.cc:227] Handling call
0810 23:32:50.917068 (+ 95939us) spinlock_profiling.cc:243] Waited 94.9 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e77d6 00007f0ce6e2d634 00007f0ce6e2d5c2 00007f0ce6e2d4f1 00007f0ce6e2d46c 00007f0ce6e2c722 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:50.917833 (+ 765us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:51.150897 (+233064us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:51.151527 (+ 630us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:51.151769 (+ 242us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:51.756262 (+604493us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:52.040523 (+284261us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:51.671461 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:51.671924 (+ 463us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:51.671950 (+ 26us) write_op.cc:435] Acquired schema lock
0810 23:32:51.671971 (+ 21us) tablet.cc:660] Decoding operations
0810 23:32:51.678208 (+ 6237us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:51.678261 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:51.678287 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:51.679089 (+ 802us) tablet.cc:699] Row locks acquired
0810 23:32:51.679113 (+ 24us) write_op.cc:260] PREPARE: finished
0810 23:32:51.679260 (+ 147us) write_op.cc:270] Start()
0810 23:32:51.679361 (+ 101us) write_op.cc:276] Timestamp: P: 1786404771679151 usec, L: 0
0810 23:32:51.679375 (+ 14us) op_driver.cc:331] REPLICATION: starting
0810 23:32:51.680052 (+ 677us) log.cc:842] Serialized 405 byte log entry
0810 23:32:51.753233 (+ 73181us) op_driver.cc:444] REPLICATION: finished
0810 23:32:51.753943 (+ 710us) write_op.cc:301] APPLY: starting
0810 23:32:51.754029 (+ 86us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:51.754152 (+ 123us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:51.754298 (+ 146us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:51.754521 (+ 223us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:51.754697 (+ 176us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:51.754747 (+ 50us) write_op.cc:312] APPLY: finished
0810 23:32:51.754998 (+ 251us) log.cc:842] Serialized 23 byte log entry
0810 23:32:51.755836 (+ 838us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:51.755909 (+ 73us) write_op.cc:454] Released schema lock
0810 23:32:51.755961 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:51.756614 (+ 653us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:51.756647 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":218216192,"child_traces":[["op",{"apply.queue_time_us":500,"apply.run_cpu_time_us":1901,"apply.run_wall_time_us":2911,"num_ops":1,"prepare.queue_time_us":518389,"prepare.run_cpu_time_us":7072,"prepare.run_wall_time_us":57325,"raft.queue_time_us":6124,"raft.run_cpu_time_us":24,"raft.run_wall_time_us":24,"replication_time_us":73745,"thread_start_us":42037,"threads_started":1}]]}
W20260810 23:32:52.098313 12240 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 35) took 1778 ms (client timeout 1000 ms). Trace:
W20260810 23:32:52.100618 12240 rpcz_store.cc:269] 0810 23:32:50.320105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.320413 (+ 308us) service_pool.cc:227] Handling call
0810 23:32:52.098221 (+1777808us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":22}
I20260810 23:32:52.104754 13514 ts_tablet_manager.cc:1428] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.160s user 0.009s sys 0.002s
W20260810 23:32:52.105896 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c244137f10db487f95395a64261b74b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:52.115522 13213 leader_election.cc:302] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:52.116677 14664 raft_consensus.cc:2726] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:52.117899 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 58387.6us
I20260810 23:32:52.121066 13213 leader_election.cc:302] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:52.120935 12229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_1841b296cea84a33bb2cc47cfb03f460" } modify_external_catalogs: true
I20260810 23:32:52.121605 12229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_1841b296cea84a33bb2cc47cfb03f460" } modify_external_catalogs: true
W20260810 23:32:52.125905 13213 leader_election.cc:341] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 124259164adf404c9cbe5f4e0d5e16c0
W20260810 23:32:52.058871 14756 log.cc:924] Time spent T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.079s user 0.000s sys 0.005s
W20260810 23:32:52.148841 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 379195us
W20260810 23:32:52.162803 13213 consensus_peers.cc:487] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13ad5d63a2104995990a86a000963cdc. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:52.164672 13974 ts_tablet_manager.cc:1918] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:52.177874 13974 log.cc:1196] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/d30bc9bd4e1f462b8d3be7f8a2f28f1e
I20260810 23:32:52.180239 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1d3bd06a9a4627a1a7e83b22dba2af with delete_type TABLET_DATA_DELETED (replaced by tablet 6596f813c59943d4a78bd813cab7aa8d) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:52.181334 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet a480093c481941f0bb35949c7606f37f with delete_type TABLET_DATA_DELETED (replaced by tablet 0703cf71490c4c71a95f55863583a03c) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:52.184055 13974 ts_tablet_manager.cc:1939] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:52.183835 13008 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:51 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:52.187528 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 187) took 5234 ms. Trace:
W20260810 23:32:52.147915 12952 leader_election.cc:341] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:52.193789 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 47645.1us
I20260810 23:32:52.197718 13327 raft_consensus.cc:1261] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 2 FOLLOWER]: Refusing update from remote peer f26517150f1c480a877f15aec71d020b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260810 23:32:52.083722 12224 rpcz_store.cc:269] 0810 23:32:50.193772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.237830 (+ 44058us) service_pool.cc:227] Handling call
0810 23:32:50.261523 (+ 23693us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:52.082644 (+1821121us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:51.133625 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:51.134581 (+ 956us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:51.134638 (+ 57us) write_op.cc:435] Acquired schema lock
0810 23:32:51.134684 (+ 46us) tablet.cc:660] Decoding operations
0810 23:32:51.163739 (+ 29055us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:51.163861 (+ 122us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:51.163967 (+ 106us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:51.164334 (+ 367us) tablet.cc:699] Row locks acquired
0810 23:32:51.164379 (+ 45us) write_op.cc:260] PREPARE: finished
0810 23:32:51.164628 (+ 249us) write_op.cc:270] Start()
0810 23:32:51.164852 (+ 224us) write_op.cc:276] Timestamp: P: 1786404771164551 usec, L: 0
0810 23:32:51.164876 (+ 24us) op_driver.cc:331] REPLICATION: starting
0810 23:32:51.296753 (+131877us) log.cc:842] Serialized 356 byte log entry
0810 23:32:51.310147 (+ 13394us) op_driver.cc:444] REPLICATION: finished
0810 23:32:51.310648 (+ 501us) write_op.cc:301] APPLY: starting
0810 23:32:51.310717 (+ 69us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:51.310838 (+ 121us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:51.310961 (+ 123us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:51.311186 (+ 225us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:51.311332 (+ 146us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:51.311381 (+ 49us) write_op.cc:312] APPLY: finished
0810 23:32:51.311622 (+ 241us) log.cc:842] Serialized 23 byte log entry
0810 23:32:51.311801 (+ 179us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:51.311880 (+ 79us) write_op.cc:454] Released schema lock
0810 23:32:51.311913 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:51.312151 (+ 238us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:51.312187 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4994,"reported_tablets":6,"spinlock_wait_cycles":36706560,"child_traces":[["op",{"apply.queue_time_us":322,"apply.run_cpu_time_us":1720,"apply.run_wall_time_us":1718,"mutex_wait_us":22162,"num_ops":1,"prepare.queue_time_us":381071,"prepare.run_cpu_time_us":4864,"prepare.run_wall_time_us":191780,"raft.queue_time_us":22639,"raft.run_cpu_time_us":881,"raft.run_wall_time_us":891,"replication_time_us":145168}]]}
W20260810 23:32:52.204674 12215 consensus_peers.cc:487] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 -> Peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Couldn't send request to peer 198f534261994047a68e1d509bb2fd8b. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260810 23:32:52.217100 12700 leader_election.cc:341] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:52.217903 12700 leader_election.cc:302] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:52.187956 13974 rpcz_store.cc:276] 0810 23:32:46.952728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:47.056068 (+103340us) service_pool.cc:227] Handling call
0810 23:32:51.181255 (+4125187us) ts_tablet_manager.cc:1145] Deleting tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e
0810 23:32:51.181303 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:51.830765 (+649462us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:52.139862 (+309097us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:52.187360 (+ 47498us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4124141}
I20260810 23:32:52.218607 12223 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_7751e2b5b7db4c62beba2c5564a6affa" } new_table_name: "default.Table_763854d208924ff78919903a71a7c826" modify_external_catalogs: true
I20260810 23:32:52.219830 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e (table default.Table_730f4da35e68448585507866c203a391 [id=c48d5d1f759b42f3a5c5a6c64588b30f]) successfully deleted
W20260810 23:32:52.220096 12700 leader_election.cc:341] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:52.220544 12700 leader_election.cc:302] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:52.220676 14678 raft_consensus.cc:2726] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:52.222186 14678 raft_consensus.cc:2726] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:52.224903 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0dd19112504f8ca57f64dcf0bd50a8" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:52.227411 12803 raft_consensus.cc:3037] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:52.231294 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet fd9d90e1ceab443dbec12b2c81b975e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:52.233081 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet 423e8d583d704fa9834026ebcd57e17e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:52.234089 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet 874ca0c3586b41a8a3074d8988813633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:52.235358 14427 consensus_queue.cc:1035] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:52.239883 13665 ts_tablet_manager.cc:1428] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.397s user 0.007s sys 0.001s
I20260810 23:32:52.241307 13665 tablet_bootstrap.cc:492] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:52.248872 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 207) took 3049 ms. Trace:
I20260810 23:32:52.249305 13935 rpcz_store.cc:276] 0810 23:32:49.198991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.199500 (+ 509us) service_pool.cc:227] Handling call
0810 23:32:52.044751 (+2845251us) ts_tablet_manager.cc:1145] Deleting tablet b261673faa964b65af3576c89e302864
0810 23:32:52.248498 (+203747us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.248773 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2836788}
I20260810 23:32:52.249893 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c29b11192604ac18c21b7bdc0238fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:38 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:52.249894 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet 440de7d9de8d4be9a5322e1fe3388b9b with delete_type TABLET_DATA_DELETED (replaced by tablet e62dc27b59ba469d99e8d11372aceff5) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:52.250638 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 205) took 3052 ms. Trace:
I20260810 23:32:52.250973 13935 rpcz_store.cc:276] 0810 23:32:49.198159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.202362 (+ 4203us) service_pool.cc:227] Handling call
0810 23:32:52.250326 (+3047964us) ts_tablet_manager.cc:1145] Deleting tablet b261673faa964b65af3576c89e302864
0810 23:32:52.250374 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.250572 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3041465}
W20260810 23:32:52.254915 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet b261673faa964b65af3576c89e302864 because the tablet was not found. No further retry: Not found: Tablet not found: b261673faa964b65af3576c89e302864
W20260810 23:32:52.255867 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet b261673faa964b65af3576c89e302864 because the tablet was not found. No further retry: Not found: Tablet not found: b261673faa964b65af3576c89e302864
I20260810 23:32:52.252002 13935 tablet_replica.cc:330] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:52.252940 13514 tablet_bootstrap.cc:492] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:52.275238 13514 tablet_bootstrap.cc:654] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:52.276353 14216 raft_consensus.cc:482] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:32:52.280125 13212 leader_election.cc:341] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Not found: Tablet not found: 124259164adf404c9cbe5f4e0d5e16c0
I20260810 23:32:52.280675 13212 leader_election.cc:302] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:52.282168 12224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_00897e6950fb4847a8b9d4c8f709cc50" } modify_external_catalogs: true
I20260810 23:32:52.282852 12224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_00897e6950fb4847a8b9d4c8f709cc50" } modify_external_catalogs: true
I20260810 23:32:52.283547 14688 raft_consensus.cc:2726] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:52.293172 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 97248.2us
I20260810 23:32:52.298821 14664 raft_consensus.cc:2726] T 4b71d565669f40d8a6d06ab267069430 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:52.287850 13691 tablet_bootstrap.cc:654] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:52.300326 12235 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_04d3b5cee61447489f042ab191392be5" } new_table_name: "default.Table_2c0b3093cb6e4b0f9e0fc90cff7fffbc" modify_external_catalogs: true
I20260810 23:32:52.297530 14121 ts_tablet_manager.cc:1918] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:52.312422 14121 log.cc:1196] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/4d697e7533af4fc0a15cb41f3fcce4db
I20260810 23:32:52.317409 14121 ts_tablet_manager.cc:1939] T 4d697e7533af4fc0a15cb41f3fcce4db P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:52.239444 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 188) took 5065 ms. Trace:
I20260810 23:32:52.320920 13974 rpcz_store.cc:276] 0810 23:32:47.174292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:47.176420 (+ 2128us) service_pool.cc:227] Handling call
0810 23:32:52.238638 (+5062218us) ts_tablet_manager.cc:1145] Deleting tablet 4cdec13ee70d402fa3a08119e7bf1313
0810 23:32:52.238779 (+ 141us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.239350 (+ 571us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5052724}
I20260810 23:32:52.323001 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 189) took 4946 ms. Trace:
W20260810 23:32:52.324604 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 4cdec13ee70d402fa3a08119e7bf1313 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cdec13ee70d402fa3a08119e7bf1313 already in progress: creating tablet
I20260810 23:32:52.313839 12803 raft_consensus.cc:2454] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
W20260810 23:32:52.329182 12958 consensus_peers.cc:487] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b261673faa964b65af3576c89e302864. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:52.324630 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 104621us
I20260810 23:32:52.300998 14216 raft_consensus.cc:504] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.335049 14216 leader_election.cc:288] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
W20260810 23:32:52.337579 12287 debug-util.cc:398] Leaking SignalData structure 0x7b08000b58c0 after lost signal to thread 12189
I20260810 23:32:52.336894 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fef127263f1470ea1056a454971e465" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:52.338444 12287 debug-util.cc:398] Leaking SignalData structure 0x7b08000c12e0 after lost signal to thread 12290
W20260810 23:32:52.340197 12952 leader_election.cc:341] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:52.343219 13691 tablet_bootstrap.cc:492] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:52.344800 12236 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 6a3a834d77474c3d8feaa5a01f3faff3 (table default.Table_7221fbab5f0b4fa892342870790a2f7e [id=1ea3dfab929845f89d4680129ae45d94]). Expected version 2 got 1
I20260810 23:32:52.345966 14798 tablet.cc:1722] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:52.344923 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0dd19112504f8ca57f64dcf0bd50a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:52.353296 13691 ts_tablet_manager.cc:1397] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.382s user 0.020s sys 0.006s
I20260810 23:32:52.363430 13336 raft_consensus.cc:3037] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:52.323343 13974 rpcz_store.cc:276] 0810 23:32:47.376836 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:47.378209 (+ 1373us) service_pool.cc:227] Handling call
0810 23:32:52.322295 (+4944086us) ts_tablet_manager.cc:1145] Deleting tablet f47b8b57d5b44a018bc2d7b3f976d803
0810 23:32:52.322389 (+ 94us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.322932 (+ 543us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4922708}
W20260810 23:32:52.365244 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet f47b8b57d5b44a018bc2d7b3f976d803 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f47b8b57d5b44a018bc2d7b3f976d803 already in progress: creating tablet
I20260810 23:32:52.366009 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fef127263f1470ea1056a454971e465" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:52.368037 12802 raft_consensus.cc:2454] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:52.369594 12229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet d603569c54ef4ffba9941d0cc3e0f97f
I20260810 23:32:52.374056 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet d603569c54ef4ffba9941d0cc3e0f97f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:52 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:52.377943 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 200) took 3554 ms. Trace:
I20260810 23:32:52.378317 13974 rpcz_store.cc:276] 0810 23:32:48.823462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:48.824159 (+ 697us) service_pool.cc:227] Handling call
0810 23:32:52.364636 (+3540477us) ts_tablet_manager.cc:1145] Deleting tablet 62705f99687340aaa1aca0942b28487f
0810 23:32:52.377019 (+ 12383us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.377822 (+ 803us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3539524}
W20260810 23:32:52.379875 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 62705f99687340aaa1aca0942b28487f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62705f99687340aaa1aca0942b28487f already in progress: creating tablet
I20260810 23:32:52.380875 13276 tablet_service.cc:1514] Processing DeleteTablet for tablet d603569c54ef4ffba9941d0cc3e0f97f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:52 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:52.391849 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_95b3b58c2c4948c199a9b573367fa598"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:52.391849 12229 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_b138f739a5db4c59af5e8e17ae77a665"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:52.390621 12958 leader_election.cc:302] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:52.395144 14427 raft_consensus.cc:2781] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:52.395779 14427 raft_consensus.cc:482] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:52.396425 14427 raft_consensus.cc:3037] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:52.403617 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 205) took 3212 ms. Trace:
I20260810 23:32:52.409288 13974 rpcz_store.cc:276] 0810 23:32:49.190850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.204158 (+ 13308us) service_pool.cc:227] Handling call
0810 23:32:52.403029 (+3198871us) ts_tablet_manager.cc:1145] Deleting tablet fd110664bac34f52bfe922a3a76c6290
0810 23:32:52.403128 (+ 99us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.403535 (+ 407us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3198001}
W20260810 23:32:52.416247 12218 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 4d697e7533af4fc0a15cb41f3fcce4db (table default.Table_26bf2c7d86704b5b981ecebb4e90ba75 [id=cefe7b97c1db42ef8712bf3c7a359af2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4d697e7533af4fc0a15cb41f3fcce4db consensus state has no leader
I20260810 23:32:52.418102 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_b3a3530a31c5486b9502e2fbba173887"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:52.420135 12229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b138f739a5db4c59af5e8e17ae77a665 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:52.420794 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3a3530a31c5486b9502e2fbba173887 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:52.424172 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95b3b58c2c4948c199a9b573367fa598 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:52.433952 12958 leader_election.cc:302] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:32:52.415534 12228 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 36) took 893 ms (client timeout 1000 ms). Trace:
I20260810 23:32:52.436607 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet d603569c54ef4ffba9941d0cc3e0f97f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:52 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:52.349654 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 154) took 7153 ms. Trace:
I20260810 23:32:52.437304 14216 raft_consensus.cc:2781] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:52.437222 14121 rpcz_store.cc:276] 0810 23:32:45.195848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.207012 (+ 11164us) service_pool.cc:227] Handling call
0810 23:32:51.505175 (+6298163us) ts_tablet_manager.cc:1145] Deleting tablet 4d697e7533af4fc0a15cb41f3fcce4db
0810 23:32:51.505245 (+ 70us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:51.678115 (+172870us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:52.297169 (+619054us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:52.349577 (+ 52408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6262072}
I20260810 23:32:52.438201 14216 raft_consensus.cc:686] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:52.453104 13935 raft_consensus.cc:2229] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:52.453701 13935 raft_consensus.cc:2258] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:52.453677 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet fd110664bac34f52bfe922a3a76c6290 because the tablet was not found. No further retry: Not found: Tablet not found: fd110664bac34f52bfe922a3a76c6290
I20260810 23:32:52.455576 13935 ts_tablet_manager.cc:1905] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:52.458129 14216 consensus_queue.cc:237] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.438835 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 4d697e7533af4fc0a15cb41f3fcce4db (table default.Table_26bf2c7d86704b5b981ecebb4e90ba75 [id=cefe7b97c1db42ef8712bf3c7a359af2]) successfully deleted
W20260810 23:32:52.435561 12228 rpcz_store.cc:269] 0810 23:32:51.504586 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:51.537187 (+ 32601us) service_pool.cc:227] Handling call
0810 23:32:52.397556 (+860369us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:52.069593 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:52.070657 (+ 1064us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:52.070718 (+ 61us) write_op.cc:435] Acquired schema lock
0810 23:32:52.070755 (+ 37us) tablet.cc:660] Decoding operations
0810 23:32:52.071365 (+ 610us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:52.071479 (+ 114us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:52.071549 (+ 70us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:52.071898 (+ 349us) tablet.cc:699] Row locks acquired
0810 23:32:52.071916 (+ 18us) write_op.cc:260] PREPARE: finished
0810 23:32:52.072246 (+ 330us) write_op.cc:270] Start()
0810 23:32:52.072454 (+ 208us) write_op.cc:276] Timestamp: P: 1786404772072155 usec, L: 0
0810 23:32:52.072476 (+ 22us) op_driver.cc:331] REPLICATION: starting
0810 23:32:52.073630 (+ 1154us) log.cc:842] Serialized 536 byte log entry
0810 23:32:52.176406 (+102776us) op_driver.cc:444] REPLICATION: finished
0810 23:32:52.194307 (+ 17901us) write_op.cc:301] APPLY: starting
0810 23:32:52.196963 (+ 2656us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:52.197098 (+ 135us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:52.197216 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:52.197443 (+ 227us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:52.197599 (+ 156us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:52.197645 (+ 46us) write_op.cc:312] APPLY: finished
0810 23:32:52.197862 (+ 217us) log.cc:842] Serialized 23 byte log entry
0810 23:32:52.198051 (+ 189us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:52.198125 (+ 74us) write_op.cc:454] Released schema lock
0810 23:32:52.198158 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:52.201501 (+ 3343us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:52.201528 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":14272,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":493,"apply.run_cpu_time_us":1953,"apply.run_wall_time_us":7422,"num_ops":1,"prepare.queue_time_us":180,"prepare.run_cpu_time_us":4638,"prepare.run_wall_time_us":27768,"replication_time_us":103815}]]}
I20260810 23:32:52.486933 13276 tablet_service.cc:1514] Processing DeleteTablet for tablet db363eb0f98d460eb6f5a577233d8ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:36 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:52.499583 13276 tablet_service.cc:1514] Processing DeleteTablet for tablet 440de7d9de8d4be9a5322e1fe3388b9b with delete_type TABLET_DATA_DELETED (replaced by tablet e62dc27b59ba469d99e8d11372aceff5) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:52.501811 14795 tablet.cc:1722] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:52.510993 13665 tablet_bootstrap.cc:654] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:52.520552 14801 raft_consensus.cc:482] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:52.521654 14801 raft_consensus.cc:504] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:52.544936 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 126753us
I20260810 23:32:52.567420 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01ede4b52c3498a924340b39c3c2435" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:52.569703 12698 leader_election.cc:341] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:52.576792 14827 tablet.cc:1722] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:52.549513 14821 tablet.cc:1722] T 6a3a834d77474c3d8feaa5a01f3faff3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:52.596649 13974 tablet_replica.cc:330] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:52.598344 13974 raft_consensus.cc:2229] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:52.599452 13935 ts_tablet_manager.cc:1918] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:52.599395 13974 raft_consensus.cc:2258] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:52.599978 13935 log.cc:1196] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/9c29b11192604ac18c21b7bdc0238fb3
I20260810 23:32:52.600863 13935 ts_tablet_manager.cc:1939] T 9c29b11192604ac18c21b7bdc0238fb3 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:52.602679 13974 ts_tablet_manager.cc:1905] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:52.605218 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 210) took 3191 ms. Trace:
I20260810 23:32:52.566123 14801 leader_election.cc:288] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:52.605638 13935 rpcz_store.cc:276] 0810 23:32:49.410451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.410735 (+ 284us) service_pool.cc:227] Handling call
0810 23:32:52.251823 (+2841088us) ts_tablet_manager.cc:1145] Deleting tablet 9c29b11192604ac18c21b7bdc0238fb3
0810 23:32:52.251863 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.524273 (+272410us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:52.599137 (+ 74864us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:52.601780 (+ 2643us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":187094,"tablet-delete.queue_time_us":2840394}
I20260810 23:32:52.607187 13935 tablet_replica.cc:330] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:52.459702 13691 raft_consensus.cc:348] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.607878 13691 raft_consensus.cc:374] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:52.608186 13935 raft_consensus.cc:2229] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:52.608275 13691 raft_consensus.cc:729] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:52.608851 13935 raft_consensus.cc:2258] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:52.609143 13691 consensus_queue.cc:260] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.611275 13935 ts_tablet_manager.cc:1905] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:52.465911 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 158) took 7016 ms. Trace:
I20260810 23:32:52.505548 14427 raft_consensus.cc:504] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.612164 14121 rpcz_store.cc:276] 0810 23:32:45.449825 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.449987 (+ 162us) service_pool.cc:227] Handling call
0810 23:32:52.438509 (+6988522us) ts_tablet_manager.cc:1145] Deleting tablet 81e26fe4d42445279f12bb17b2841d08
0810 23:32:52.465129 (+ 26620us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.465697 (+ 568us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6987583}
I20260810 23:32:52.613611 14121 tablet_replica.cc:330] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:52.614632 14427 leader_election.cc:288] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:52.616312 13691 ts_tablet_manager.cc:1428] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.263s user 0.010s sys 0.000s
I20260810 23:32:52.617563 14121 raft_consensus.cc:2229] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:52.618264 14121 raft_consensus.cc:2258] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:52.619553 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fef127263f1470ea1056a454971e465" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:52.620409 12805 raft_consensus.cc:3037] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:52.620901 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fef127263f1470ea1056a454971e465" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
W20260810 23:32:52.625957 12952 consensus_peers.cc:487] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:52.626797 12952 leader_election.cc:341] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:52.628446 13691 tablet_bootstrap.cc:492] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:52.552580 14823 tablet.cc:1722] T 6a3a834d77474c3d8feaa5a01f3faff3 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:52.565704 14792 consensus_queue.cc:1035] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20260810 23:32:52.576471 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01ede4b52c3498a924340b39c3c2435" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:52.568653 14825 tablet.cc:1722] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260810 23:32:52.633996 12700 leader_election.cc:341] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:52.562896 14819 tablet.cc:1722] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:52.634559 12700 leader_election.cc:302] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:52.636261 14801 raft_consensus.cc:2726] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:52.639429 14792 raft_consensus.cc:482] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:52.575479 12237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e0909dd9de4c461dae94163efea3a62f" } modify_external_catalogs: true
I20260810 23:32:52.640775 12237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e0909dd9de4c461dae94163efea3a62f" } modify_external_catalogs: true
I20260810 23:32:52.640463 14792 raft_consensus.cc:504] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:52.615221 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 81e26fe4d42445279f12bb17b2841d08 because the tablet was not found. No further retry: Not found: Tablet not found: 81e26fe4d42445279f12bb17b2841d08
I20260810 23:32:52.645011 14792 leader_election.cc:288] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:52.645507 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10033af79e854103bc47a046d2219ca0" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:52.646687 12806 raft_consensus.cc:2454] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:52.647306 13276 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d697e7533af4fc0a15cb41f3fcce4db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:52.648339 12958 leader_election.cc:302] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:52.632798 14121 ts_tablet_manager.cc:1905] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:52.669835 12805 raft_consensus.cc:2454] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:52.679636 12958 leader_election.cc:302] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:52.688995 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_9bbb5f34eb6d4d65995978e1c125a8bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:52.689929 13665 tablet_bootstrap.cc:492] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:52.690682 13665 ts_tablet_manager.cc:1397] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.450s user 0.014s sys 0.007s
W20260810 23:32:52.691637 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bbb5f34eb6d4d65995978e1c125a8bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:52.701072 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10033af79e854103bc47a046d2219ca0" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:52.719725 13935 ts_tablet_manager.cc:1918] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:52.726547 13665 raft_consensus.cc:348] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.728188 13665 raft_consensus.cc:374] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:52.731186 14427 raft_consensus.cc:2781] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:52.731637 14427 raft_consensus.cc:482] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:52.731959 14427 raft_consensus.cc:3037] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:52.732335 13665 raft_consensus.cc:729] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
W20260810 23:32:52.737253 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 57739us
I20260810 23:32:52.738866 14792 raft_consensus.cc:2781] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:52.739719 14792 raft_consensus.cc:686] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:52.740249 13935 log.cc:1196] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/423e8d583d704fa9834026ebcd57e17e
I20260810 23:32:52.738659 13665 consensus_queue.cc:260] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.740859 14792 consensus_queue.cc:237] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.743379 13665 ts_tablet_manager.cc:1428] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.052s user 0.006s sys 0.003s
I20260810 23:32:52.744196 13935 ts_tablet_manager.cc:1939] T 423e8d583d704fa9834026ebcd57e17e P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
W20260810 23:32:52.748832 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 77194.1us
W20260810 23:32:52.749874 12952 leader_election.cc:341] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:52.754562 13665 tablet_bootstrap.cc:492] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:52.782208 13514 tablet_bootstrap.cc:492] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:52.782850 13514 ts_tablet_manager.cc:1397] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.530s user 0.008s sys 0.015s
I20260810 23:32:52.784452 13665 tablet_bootstrap.cc:654] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:52.791162 12215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20260810 23:32:52.796994 14815 log.cc:924] Time spent T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
W20260810 23:32:52.811892 12958 consensus_peers.cc:487] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c29b11192604ac18c21b7bdc0238fb3. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:52.824779 12235 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba8d56858471421a9fbc361ee6aff28f consensus state has no leader
I20260810 23:32:52.826048 12235 master_service.cc:946] ReplaceTablet: received request to replace tablet 240adfa4be844518bbc9624d0e408789 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:52.837720 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 209) took 3427 ms. Trace:
I20260810 23:32:52.848531 13935 rpcz_store.cc:276] 0810 23:32:49.410095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.412547 (+ 2452us) service_pool.cc:227] Handling call
0810 23:32:52.606976 (+3194429us) ts_tablet_manager.cc:1145] Deleting tablet 423e8d583d704fa9834026ebcd57e17e
0810 23:32:52.607036 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.675711 (+ 68675us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:52.719442 (+ 43731us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:52.837621 (+118179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3193832}
I20260810 23:32:52.850371 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 208) took 3440 ms. Trace:
I20260810 23:32:52.850901 13935 rpcz_store.cc:276] 0810 23:32:49.409713 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.441850 (+ 32137us) service_pool.cc:227] Handling call
0810 23:32:52.849955 (+3408105us) ts_tablet_manager.cc:1145] Deleting tablet 4d697e7533af4fc0a15cb41f3fcce4db
0810 23:32:52.850017 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.850277 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3396776}
I20260810 23:32:52.853148 13691 tablet_bootstrap.cc:700] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b: Time spent opening tablet: real 0.198s user 0.004s sys 0.000s
I20260810 23:32:52.854110 13691 tablet_bootstrap.cc:654] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:52.854139 13935 tablet_replica.cc:330] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:52.855464 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 9c29b11192604ac18c21b7bdc0238fb3 (table default.Table_dd2cd30e5e2349de9180be01866dc513 [id=8754da7be72b4eef80daacdcdfde82a7]) successfully deleted
I20260810 23:32:52.855515 13935 raft_consensus.cc:2229] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:52.856565 13935 raft_consensus.cc:2258] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:52.862807 12958 consensus_peers.cc:487] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 423e8d583d704fa9834026ebcd57e17e. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:52.862803 13935 ts_tablet_manager.cc:1905] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:52.868719 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 423e8d583d704fa9834026ebcd57e17e (table default.Table_514d038dba3a4fc399b97e7edc254ad9 [id=53909ddaaede4deb97cec7ad02020c5d]) successfully deleted
W20260810 23:32:52.874854 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 4d697e7533af4fc0a15cb41f3fcce4db because the tablet was not found. No further retry: Not found: Tablet not found: 4d697e7533af4fc0a15cb41f3fcce4db
I20260810 23:32:52.876958 13665 tablet_bootstrap.cc:492] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:52.877579 13665 ts_tablet_manager.cc:1397] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.123s user 0.013s sys 0.015s
I20260810 23:32:52.918870 13665 raft_consensus.cc:348] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.924933 14427 raft_consensus.cc:504] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.929932 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10033af79e854103bc47a046d2219ca0" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
W20260810 23:32:52.933202 12952 leader_election.cc:341] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:52.939963 14841 tablet.cc:1722] T 6a3a834d77474c3d8feaa5a01f3faff3 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:32:52.956388 13665 raft_consensus.cc:374] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:52.956808 13665 raft_consensus.cc:729] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:52.958755 12235 master_service.cc:946] ReplaceTablet: received request to replace tablet 4f453bd72168429086837b2b9b09fc78 from {username='slave'} at 127.0.0.1:34742
W20260810 23:32:52.971300 12223 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea consensus state has no leader
I20260810 23:32:52.959990 13665 consensus_queue.cc:260] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:52.996400 14427 leader_election.cc:288] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:53.005914 14121 ts_tablet_manager.cc:1918] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:53.006428 14121 log.cc:1196] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/fd4f55ef709d473f89a16d32160eb61b
I20260810 23:32:53.007146 14121 ts_tablet_manager.cc:1939] T fd4f55ef709d473f89a16d32160eb61b P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:53.025997 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 159) took 7405 ms. Trace:
I20260810 23:32:53.026413 14121 rpcz_store.cc:276] 0810 23:32:45.621113 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:45.621516 (+ 403us) service_pool.cc:227] Handling call
0810 23:32:52.613334 (+6991818us) ts_tablet_manager.cc:1145] Deleting tablet fd4f55ef709d473f89a16d32160eb61b
0810 23:32:52.613457 (+ 123us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.727945 (+114488us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.005614 (+277669us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.025905 (+ 20291us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2899,"tablet-delete.queue_time_us":6932947}
I20260810 23:32:53.027946 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet fd4f55ef709d473f89a16d32160eb61b (table default.Table_addbc3e258944d0cb5d2e7fb11214412 [id=d4e32fc5299547268eb0c93e2d3c00e1]) successfully deleted
I20260810 23:32:53.030555 13665 ts_tablet_manager.cc:1428] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.153s user 0.015s sys 0.002s
I20260810 23:32:53.032989 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 164) took 6886 ms. Trace:
I20260810 23:32:53.033334 14121 rpcz_store.cc:276] 0810 23:32:46.146845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:46.196597 (+ 49752us) service_pool.cc:227] Handling call
0810 23:32:53.032378 (+6835781us) ts_tablet_manager.cc:1145] Deleting tablet 80b54edbb45d496c94efedb6feee8e45
0810 23:32:53.032564 (+ 186us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.032913 (+ 349us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6826086}
W20260810 23:32:53.034653 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 80b54edbb45d496c94efedb6feee8e45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80b54edbb45d496c94efedb6feee8e45 already in progress: creating tablet
I20260810 23:32:53.040475 14121 tablet_replica.cc:330] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:53.042495 14121 raft_consensus.cc:2229] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:53.043852 14121 raft_consensus.cc:2258] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:53.046759 14121 ts_tablet_manager.cc:1905] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:53.054143 13213 consensus_peers.cc:487] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:53.055279 13665 tablet_bootstrap.cc:492] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:53.060043 13691 tablet_bootstrap.cc:492] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:53.060770 13691 ts_tablet_manager.cc:1397] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.433s user 0.017s sys 0.003s
I20260810 23:32:52.867553 13514 raft_consensus.cc:348] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:53.008247 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10033af79e854103bc47a046d2219ca0" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:53.065157 12802 raft_consensus.cc:3037] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:53.068228 13514 raft_consensus.cc:374] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:53.071568 14121 ts_tablet_manager.cc:1918] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:53.072220 13514 raft_consensus.cc:729] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:53.073097 14121 log.cc:1196] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/d30bc9bd4e1f462b8d3be7f8a2f28f1e
I20260810 23:32:53.073473 13514 consensus_queue.cc:260] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:53.074010 14121 ts_tablet_manager.cc:1939] T d30bc9bd4e1f462b8d3be7f8a2f28f1e P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:53.077860 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 173) took 6093 ms. Trace:
I20260810 23:32:53.078213 14121 rpcz_store.cc:276] 0810 23:32:46.985244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:46.988905 (+ 3661us) service_pool.cc:227] Handling call
0810 23:32:53.040261 (+6051356us) ts_tablet_manager.cc:1145] Deleting tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e
0810 23:32:53.040323 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.061687 (+ 21364us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.071286 (+ 9599us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.077789 (+ 6503us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5921705}
I20260810 23:32:53.079593 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e (table default.Table_730f4da35e68448585507866c203a391 [id=c48d5d1f759b42f3a5c5a6c64588b30f]) successfully deleted
I20260810 23:32:53.080207 12802 raft_consensus.cc:2454] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:53.081419 12958 leader_election.cc:302] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:53.082600 14427 raft_consensus.cc:2781] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:53.008600 14795 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:47546 (request call id 228) took 1640 ms. Trace:
I20260810 23:32:53.083093 14427 raft_consensus.cc:686] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:53.083994 14427 consensus_queue.cc:237] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:53.083073 14795 rpcz_store.cc:276] 0810 23:32:51.368703 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:51.368889 (+ 186us) service_pool.cc:227] Handling call
0810 23:32:53.008520 (+1639631us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:51.370244 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 23:32:51.370647 (+ 403us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 23:32:51.370683 (+ 36us) alter_schema_op.cc:65] Acquired schema lock
0810 23:32:51.370766 (+ 83us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 23:32:51.370930 (+ 164us) alter_schema_op.cc:126] START. Timestamp: 1786404771370533
0810 23:32:51.370960 (+ 30us) op_driver.cc:331] REPLICATION: starting
0810 23:32:51.371589 (+ 629us) log.cc:842] Serialized 173 byte log entry
0810 23:32:52.128354 (+756765us) op_driver.cc:444] REPLICATION: finished
0810 23:32:52.501665 (+373311us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 23:32:52.544256 (+ 42591us) log.cc:842] Serialized 5 byte log entry
0810 23:32:52.892330 (+348074us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.006248 (+113918us) log.cc:842] Serialized 14 byte log entry
0810 23:32:53.008320 (+ 2072us) alter_schema_op.cc:71] Released schema lock
0810 23:32:53.008357 (+ 37us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":373005,"prepare.queue_time_us":216,"prepare.run_cpu_time_us":20360,"prepare.run_wall_time_us":206151,"replication_time_us":757245,"thread_start_us":575753,"threads_started":2,"wal-append.queue_time_us":228672}]]}
I20260810 23:32:53.089497 13514 ts_tablet_manager.cc:1428] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.306s user 0.005s sys 0.006s
I20260810 23:32:53.090291 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 174) took 5863 ms. Trace:
I20260810 23:32:53.090667 14121 rpcz_store.cc:276] 0810 23:32:47.217621 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:47.224656 (+ 7035us) service_pool.cc:227] Handling call
0810 23:32:53.080625 (+5855969us) ts_tablet_manager.cc:1145] Deleting tablet 4cdec13ee70d402fa3a08119e7bf1313
0810 23:32:53.080687 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.081041 (+ 354us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5775136}
I20260810 23:32:53.094151 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 175) took 5714 ms. Trace:
I20260810 23:32:53.094558 14121 rpcz_store.cc:276] 0810 23:32:47.380254 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:47.384140 (+ 3886us) service_pool.cc:227] Handling call
0810 23:32:53.093621 (+5709481us) ts_tablet_manager.cc:1145] Deleting tablet f47b8b57d5b44a018bc2d7b3f976d803
0810 23:32:53.093686 (+ 65us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.094064 (+ 378us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5708494}
W20260810 23:32:53.114516 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet f47b8b57d5b44a018bc2d7b3f976d803 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f47b8b57d5b44a018bc2d7b3f976d803 already in progress: creating tablet
I20260810 23:32:53.116391 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 185) took 4282 ms. Trace:
I20260810 23:32:53.116838 14121 rpcz_store.cc:276] 0810 23:32:48.833950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:48.834155 (+ 205us) service_pool.cc:227] Handling call
0810 23:32:53.115896 (+4281741us) ts_tablet_manager.cc:1145] Deleting tablet 62705f99687340aaa1aca0942b28487f
0810 23:32:53.115956 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.116322 (+ 366us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4281002}
W20260810 23:32:53.121189 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 62705f99687340aaa1aca0942b28487f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62705f99687340aaa1aca0942b28487f already in progress: creating tablet
I20260810 23:32:53.122872 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 192) took 3825 ms. Trace:
I20260810 23:32:53.123219 14121 rpcz_store.cc:276] 0810 23:32:49.296976 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.297208 (+ 232us) service_pool.cc:227] Handling call
0810 23:32:53.122527 (+3825319us) ts_tablet_manager.cc:1145] Deleting tablet a0ba75f2c2b54ec1968db8feeacf8383
0810 23:32:53.122578 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.122793 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3824448}
I20260810 23:32:53.124166 12223 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_b8310838312249ab80ce8a196b1c354f" } new_table_name: "default.Table_2f4392eef1c74daa870d52081ffb3c90" modify_external_catalogs: true
W20260810 23:32:53.124533 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet a0ba75f2c2b54ec1968db8feeacf8383 because the tablet was not found. No further retry: Not found: Tablet not found: a0ba75f2c2b54ec1968db8feeacf8383
I20260810 23:32:53.127367 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 193) took 3829 ms. Trace:
I20260810 23:32:53.127717 14121 rpcz_store.cc:276] 0810 23:32:49.297822 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.298271 (+ 449us) service_pool.cc:227] Handling call
0810 23:32:53.127035 (+3828764us) ts_tablet_manager.cc:1145] Deleting tablet 18b29cda94584badb1225477ec2bd1bc
0810 23:32:53.127085 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.127301 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3827053}
W20260810 23:32:53.130803 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 18b29cda94584badb1225477ec2bd1bc because the tablet was not found. No further retry: Not found: Tablet not found: 18b29cda94584badb1225477ec2bd1bc
I20260810 23:32:53.132383 13665 tablet_bootstrap.cc:654] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:53.132925 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 194) took 3825 ms. Trace:
I20260810 23:32:53.133255 14121 rpcz_store.cc:276] 0810 23:32:49.307791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.307944 (+ 153us) service_pool.cc:227] Handling call
0810 23:32:53.132346 (+3824402us) ts_tablet_manager.cc:1145] Deleting tablet 124259164adf404c9cbe5f4e0d5e16c0
0810 23:32:53.132650 (+ 304us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.132858 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3823701}
W20260810 23:32:53.134578 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 124259164adf404c9cbe5f4e0d5e16c0 because the tablet was not found. No further retry: Not found: Tablet not found: 124259164adf404c9cbe5f4e0d5e16c0
I20260810 23:32:53.137549 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 195) took 3828 ms. Trace:
I20260810 23:32:53.137869 14121 rpcz_store.cc:276] 0810 23:32:49.308551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.308685 (+ 134us) service_pool.cc:227] Handling call
0810 23:32:53.137135 (+3828450us) ts_tablet_manager.cc:1145] Deleting tablet b8986e59b2314bd69b9b1de36f1f2ba7
0810 23:32:53.137185 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.137383 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3827844}
W20260810 23:32:53.140997 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet b8986e59b2314bd69b9b1de36f1f2ba7 because the tablet was not found. No further retry: Not found: Tablet not found: b8986e59b2314bd69b9b1de36f1f2ba7
I20260810 23:32:53.142294 14121 tablet_replica.cc:330] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
W20260810 23:32:53.100888 12698 consensus_peers.cc:487] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:53.144973 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 44262us
I20260810 23:32:53.141398 14814 consensus_queue.cc:1035] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260810 23:32:53.091939 13935 ts_tablet_manager.cc:1918] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260810 23:32:53.147893 13935 log.cc:1196] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/08c45b71432a45269cd6a106fc5f7292
I20260810 23:32:53.155118 13514 tablet_bootstrap.cc:492] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:53.161923 14121 raft_consensus.cc:2229] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 2 LEADER]: Raft consensus shutting down.
I20260810 23:32:53.183571 14121 raft_consensus.cc:2258] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:53.188277 14121 ts_tablet_manager.cc:1905] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:53.193405 14427 consensus_queue.cc:1035] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:53.164359 13935 ts_tablet_manager.cc:1939] T 08c45b71432a45269cd6a106fc5f7292 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:53.239324 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 213) took 3660 ms. Trace:
I20260810 23:32:53.240463 13935 rpcz_store.cc:276] 0810 23:32:49.578599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.578723 (+ 124us) service_pool.cc:227] Handling call
0810 23:32:52.853884 (+3275161us) ts_tablet_manager.cc:1145] Deleting tablet 08c45b71432a45269cd6a106fc5f7292
0810 23:32:52.853950 (+ 66us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.943043 (+ 89093us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.091598 (+148555us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.239239 (+147641us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3264390}
I20260810 23:32:53.250521 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 212) took 3672 ms. Trace:
I20260810 23:32:53.250463 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_75ca982c15e34c8d9603f100dff5b392 [id=b809029bf6cc48ab917e48bbf2109ae1]) successfully deleted
I20260810 23:32:53.250891 13935 rpcz_store.cc:276] 0810 23:32:49.578232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.579231 (+ 999us) service_pool.cc:227] Handling call
0810 23:32:53.250042 (+3670811us) ts_tablet_manager.cc:1145] Deleting tablet 240adfa4be844518bbc9624d0e408789
0810 23:32:53.250100 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.250436 (+ 336us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3662084}
W20260810 23:32:53.257477 12952 consensus_peers.cc:487] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:53.063845 13691 raft_consensus.cc:348] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:53.260355 13665 tablet_bootstrap.cc:492] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:53.260339 13691 raft_consensus.cc:374] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:53.261083 13665 ts_tablet_manager.cc:1397] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.215s user 0.013s sys 0.003s
I20260810 23:32:53.260989 13691 raft_consensus.cc:729] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:53.261089 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 220) took 3024 ms. Trace:
I20260810 23:32:53.261439 13935 rpcz_store.cc:276] 0810 23:32:50.236988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.237474 (+ 486us) service_pool.cc:227] Handling call
0810 23:32:53.251862 (+3014388us) ts_tablet_manager.cc:1145] Deleting tablet 4d0d9227d8574de8bd73041f3e8043a0
0810 23:32:53.251911 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.260969 (+ 9058us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3013204}
I20260810 23:32:53.262852 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 221) took 2966 ms. Trace:
W20260810 23:32:53.263142 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 4d0d9227d8574de8bd73041f3e8043a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d0d9227d8574de8bd73041f3e8043a0 already in progress: creating tablet
I20260810 23:32:53.263171 13935 rpcz_store.cc:276] 0810 23:32:50.296422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.296713 (+ 291us) service_pool.cc:227] Handling call
0810 23:32:53.262475 (+2965762us) ts_tablet_manager.cc:1145] Deleting tablet f1e8311c941b43058384cee6d8263dc1
0810 23:32:53.262522 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.262792 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2964266}
W20260810 23:32:53.264596 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet f1e8311c941b43058384cee6d8263dc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e8311c941b43058384cee6d8263dc1 already in progress: creating tablet
I20260810 23:32:53.263844 13665 raft_consensus.cc:348] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:53.282529 13665 raft_consensus.cc:374] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:53.282881 13665 raft_consensus.cc:729] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:53.293720 13665 consensus_queue.cc:260] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:53.261909 13691 consensus_queue.cc:260] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:53.317121 13691 ts_tablet_manager.cc:1428] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.256s user 0.007s sys 0.000s
I20260810 23:32:53.321290 13665 ts_tablet_manager.cc:1428] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.060s user 0.006s sys 0.001s
I20260810 23:32:53.325258 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 223) took 2516 ms. Trace:
I20260810 23:32:53.325697 13935 rpcz_store.cc:276] 0810 23:32:50.785223 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.787816 (+ 2593us) service_pool.cc:227] Handling call
0810 23:32:53.264011 (+2476195us) ts_tablet_manager.cc:1145] Deleting tablet 0be35dabbec44caebd1df687b167f34a
0810 23:32:53.301053 (+ 37042us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.301440 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2475464}
W20260810 23:32:53.327406 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 0be35dabbec44caebd1df687b167f34a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0be35dabbec44caebd1df687b167f34a already in progress: creating tablet
I20260810 23:32:53.327385 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 231) took 1442 ms. Trace:
I20260810 23:32:53.331177 13665 tablet_bootstrap.cc:492] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:53.340438 13336 raft_consensus.cc:3037] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:53.352682 13514 tablet_bootstrap.cc:700] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent opening tablet: real 0.117s user 0.002s sys 0.001s
I20260810 23:32:53.327906 13935 rpcz_store.cc:276] 0810 23:32:51.884635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:51.884819 (+ 184us) service_pool.cc:227] Handling call
0810 23:32:53.326764 (+1441945us) ts_tablet_manager.cc:1145] Deleting tablet 4d1d3bd06a9a4627a1a7e83b22dba2af
0810 23:32:53.326834 (+ 70us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.327282 (+ 448us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1441158}
W20260810 23:32:53.358089 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 4d1d3bd06a9a4627a1a7e83b22dba2af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d1d3bd06a9a4627a1a7e83b22dba2af already in progress: creating tablet
I20260810 23:32:53.368316 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 230) took 1489 ms. Trace:
I20260810 23:32:53.372365 13935 rpcz_store.cc:276] 0810 23:32:51.878651 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:51.881677 (+ 3026us) service_pool.cc:227] Handling call
0810 23:32:53.367841 (+1486164us) ts_tablet_manager.cc:1145] Deleting tablet 1b1f24e226b94ccf8ad5a656c80a5b3b
0810 23:32:53.367889 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.368235 (+ 346us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1457917}
I20260810 23:32:53.374045 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 232) took 1414 ms. Trace:
W20260810 23:32:53.374230 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b1f24e226b94ccf8ad5a656c80a5b3b already in progress: creating tablet
I20260810 23:32:53.374375 13935 rpcz_store.cc:276] 0810 23:32:51.959978 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:51.962895 (+ 2917us) service_pool.cc:227] Handling call
0810 23:32:53.373626 (+1410731us) ts_tablet_manager.cc:1145] Deleting tablet a480093c481941f0bb35949c7606f37f
0810 23:32:53.373681 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.373985 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1409857}
I20260810 23:32:53.375654 13935 tablet_replica.cc:330] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
W20260810 23:32:53.376135 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet a480093c481941f0bb35949c7606f37f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a480093c481941f0bb35949c7606f37f already in progress: creating tablet
I20260810 23:32:53.353688 13514 tablet_bootstrap.cc:654] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:53.389411 13665 tablet_bootstrap.cc:654] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:53.392895 13691 tablet_bootstrap.cc:492] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:53.396447 13935 raft_consensus.cc:2229] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:53.397378 13935 raft_consensus.cc:2258] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:53.379732 13974 ts_tablet_manager.cc:1918] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:53.402848 13974 log.cc:1196] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/124259164adf404c9cbe5f4e0d5e16c0
I20260810 23:32:53.404078 13974 ts_tablet_manager.cc:1939] T 124259164adf404c9cbe5f4e0d5e16c0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:53.423214 13935 ts_tablet_manager.cc:1905] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:53.426347 12226 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_6c7fe24265af4acaab757c9857a87a7e" } new_table_name: "default.Table_dc4c377dd45047ba9e9c97d06a255575" modify_external_catalogs: true
I20260810 23:32:53.428535 13665 tablet_bootstrap.cc:492] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:53.429172 13665 ts_tablet_manager.cc:1397] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.098s user 0.013s sys 0.001s
I20260810 23:32:53.429842 14121 ts_tablet_manager.cc:1918] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260810 23:32:53.430326 14121 log.cc:1196] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/b261673faa964b65af3576c89e302864
I20260810 23:32:53.431412 14121 ts_tablet_manager.cc:1939] T b261673faa964b65af3576c89e302864 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
W20260810 23:32:53.454504 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet ba8d56858471421a9fbc361ee6aff28f (table default.Table_2c0b3093cb6e4b0f9e0fc90cff7fffbc [id=9d36464c8de043e7a91673f011908f50], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ba8d56858471421a9fbc361ee6aff28f consensus state has no leader
I20260810 23:32:53.459667 13665 raft_consensus.cc:348] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:53.463267 13665 raft_consensus.cc:374] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:53.463559 13665 raft_consensus.cc:729] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:53.468868 12229 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 300) took 1084 ms. Trace:
I20260810 23:32:53.469292 12229 rpcz_store.cc:276] 0810 23:32:52.384090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.384208 (+ 118us) service_pool.cc:227] Handling call
0810 23:32:52.391627 (+ 7419us) spinlock_profiling.cc:243] Waited 6.32 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7156 00007f0ce6e24034 00007f0ce6e23fc2 00007f0ce6e23ef1 00007f0ce6e23e6c 00007f0ce6e23122 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:52.420645 (+ 29018us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:52.435622 (+ 14977us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:53.467261 (+1031639us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:53.467645 (+ 384us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:53.467957 (+ 312us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:53.400460 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:53.400997 (+ 537us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:53.401030 (+ 33us) write_op.cc:435] Acquired schema lock
0810 23:32:53.401045 (+ 15us) tablet.cc:660] Decoding operations
0810 23:32:53.403497 (+ 2452us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:53.403554 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:53.403584 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:53.403847 (+ 263us) tablet.cc:699] Row locks acquired
0810 23:32:53.403867 (+ 20us) write_op.cc:260] PREPARE: finished
0810 23:32:53.404874 (+ 1007us) write_op.cc:270] Start()
0810 23:32:53.404992 (+ 118us) write_op.cc:276] Timestamp: P: 1786404773404802 usec, L: 0
0810 23:32:53.405008 (+ 16us) op_driver.cc:331] REPLICATION: starting
0810 23:32:53.423366 (+ 18358us) log.cc:842] Serialized 433 byte log entry
0810 23:32:53.441252 (+ 17886us) spinlock_profiling.cc:243] Waited 5.46 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:53.463568 (+ 22316us) op_driver.cc:444] REPLICATION: finished
0810 23:32:53.463996 (+ 428us) write_op.cc:301] APPLY: starting
0810 23:32:53.464150 (+ 154us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:53.464310 (+ 160us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:53.464331 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:53.465277 (+ 946us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:53.465800 (+ 523us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:53.465909 (+ 109us) write_op.cc:312] APPLY: finished
0810 23:32:53.466249 (+ 340us) log.cc:842] Serialized 29 byte log entry
0810 23:32:53.466477 (+ 228us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:53.466560 (+ 83us) write_op.cc:454] Released schema lock
0810 23:32:53.466594 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:53.466793 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:53.466814 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":14536960,"child_traces":[["op",{"apply.queue_time_us":108,"apply.run_cpu_time_us":3045,"apply.run_wall_time_us":3044,"num_ops":2,"prepare.queue_time_us":951375,"prepare.run_cpu_time_us":4056,"prepare.run_wall_time_us":41109,"raft.queue_time_us":22991,"raft.run_cpu_time_us":1236,"raft.run_wall_time_us":17114,"replication_time_us":58453,"spinlock_wait_cycles":12549376}]]}
I20260810 23:32:53.472868 13665 consensus_queue.cc:260] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:53.473480 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 208) took 4194 ms. Trace:
I20260810 23:32:53.474301 13974 rpcz_store.cc:276] 0810 23:32:49.262554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.263104 (+ 550us) service_pool.cc:227] Handling call
0810 23:32:52.494628 (+3231524us) ts_tablet_manager.cc:1145] Deleting tablet 124259164adf404c9cbe5f4e0d5e16c0
0810 23:32:52.596178 (+101550us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:52.899741 (+303563us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.379420 (+479679us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.457066 (+ 77646us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3230395}
I20260810 23:32:53.481524 13691 tablet_bootstrap.cc:654] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:53.504307 13514 tablet_bootstrap.cc:492] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:53.505127 13514 ts_tablet_manager.cc:1397] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.350s user 0.010s sys 0.007s
I20260810 23:32:53.509123 13665 ts_tablet_manager.cc:1428] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.080s user 0.009s sys 0.000s
I20260810 23:32:53.511201 13665 tablet_bootstrap.cc:492] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:53.514133 13974 tablet_replica.cc:330] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:53.515339 13974 raft_consensus.cc:2229] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:53.537458 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 124259164adf404c9cbe5f4e0d5e16c0 (table default.Table_994c5ce0643d422595ad55d7f893ae9d [id=69a8ac9bc8ae4c53aefbed949cf8ac95]) successfully deleted
I20260810 23:32:53.536886 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 196) took 4226 ms. Trace:
I20260810 23:32:53.536139 13514 raft_consensus.cc:348] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:53.538271 14121 rpcz_store.cc:276] 0810 23:32:49.310157 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.310281 (+ 124us) service_pool.cc:227] Handling call
0810 23:32:53.142114 (+3831833us) ts_tablet_manager.cc:1145] Deleting tablet b261673faa964b65af3576c89e302864
0810 23:32:53.142161 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.256808 (+114647us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.429533 (+172725us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.536791 (+107258us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18252,"tablet-delete.queue_time_us":3831257}
I20260810 23:32:53.538830 13514 raft_consensus.cc:374] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:53.539884 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 197) took 4228 ms. Trace:
I20260810 23:32:53.539808 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet b261673faa964b65af3576c89e302864 (table default.Table_adf76678e17a494185985a544def0f2d [id=5a377948adc7475c8c98141deaf1c274]) successfully deleted
I20260810 23:32:53.540196 14121 rpcz_store.cc:276] 0810 23:32:49.311831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.311934 (+ 103us) service_pool.cc:227] Handling call
0810 23:32:53.539502 (+4227568us) ts_tablet_manager.cc:1145] Deleting tablet 3d596299c82b4ff1ac151d68b97ff55b
0810 23:32:53.539545 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.539821 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4226977}
I20260810 23:32:53.540679 13514 raft_consensus.cc:729] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:53.541800 13514 consensus_queue.cc:260] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:53.548709 13974 raft_consensus.cc:2258] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:53.561594 13935 ts_tablet_manager.cc:1918] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:53.562178 13935 log.cc:1196] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/fd9d90e1ceab443dbec12b2c81b975e8
I20260810 23:32:53.562593 13514 ts_tablet_manager.cc:1428] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.057s user 0.008s sys 0.001s
I20260810 23:32:53.563200 13935 ts_tablet_manager.cc:1939] T fd9d90e1ceab443dbec12b2c81b975e8 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:53.564381 14865 mvcc.cc:205] Tried to move back new op lower bound from 7317113947952824320 to 7317113939320430592. Current Snapshot: MvccSnapshot[applied={T|T < 7317113947952824320}]
I20260810 23:32:53.569628 13974 ts_tablet_manager.cc:1905] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:53.578542 13691 tablet_bootstrap.cc:492] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:53.579747 13691 ts_tablet_manager.cc:1397] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.187s user 0.015s sys 0.000s
I20260810 23:32:53.589269 14873 tablet.cc:1722] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:53.605237 13336 raft_consensus.cc:3037] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:53.611958 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 233) took 1547 ms. Trace:
I20260810 23:32:53.614799 14427 consensus_queue.cc:1035] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:53.618309 14121 tablet_replica.cc:330] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:53.612485 13935 rpcz_store.cc:276] 0810 23:32:52.064917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.065061 (+ 144us) service_pool.cc:227] Handling call
0810 23:32:53.375468 (+1310407us) ts_tablet_manager.cc:1145] Deleting tablet fd9d90e1ceab443dbec12b2c81b975e8
0810 23:32:53.375525 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.479909 (+104384us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.561203 (+ 81294us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.611877 (+ 50674us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1309674}
I20260810 23:32:53.623415 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet fd9d90e1ceab443dbec12b2c81b975e8 (table default.Table_dc5d83ea7f1441fa9e2b2536bab49724 [id=aef6fd3aebac448e96d5942fd12836ca]) successfully deleted
I20260810 23:32:53.626427 13935 tablet_replica.cc:330] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:53.627444 13935 raft_consensus.cc:2229] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:53.627948 13514 tablet_bootstrap.cc:492] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:53.629850 13276 tablet_service.cc:1467] Processing CreateTablet for tablet d53029158b0142078d66bad842f0a69e (DEFAULT_TABLE table=default.Table_9e92a0778ad2454eb9ffe997c50710c9 [id=7cb602df18424e85a49d13761d5b5bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.631134 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53029158b0142078d66bad842f0a69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.634443 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 92b81e7056a441d2854ba5ab6b4c5433 (table default.Table_e0faeb2fda9f4af48212613473c88681 [id=d8d3c66123cd4516a59a0ab8ae4d47b9]). Expected version 2 got 0
I20260810 23:32:53.637001 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet ab8d987109ba40198645881b449cb972 (table default.Table_d8a0e3e655b9417d9df3ff6d82b76f15 [id=d0b86eafe9544cd98d8fce8ece3b5c10]). Expected version 1 got 0
W20260810 23:32:53.638291 12952 consensus_peers.cc:487] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:53.649835 14801 raft_consensus.cc:482] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:53.650599 13665 tablet_bootstrap.cc:654] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:53.655403 12764 tablet_service.cc:1467] Processing CreateTablet for tablet d53029158b0142078d66bad842f0a69e (DEFAULT_TABLE table=default.Table_9e92a0778ad2454eb9ffe997c50710c9 [id=7cb602df18424e85a49d13761d5b5bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.656447 13031 tablet_service.cc:1467] Processing CreateTablet for tablet d53029158b0142078d66bad842f0a69e (DEFAULT_TABLE table=default.Table_9e92a0778ad2454eb9ffe997c50710c9 [id=7cb602df18424e85a49d13761d5b5bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.657191 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53029158b0142078d66bad842f0a69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.658149 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53029158b0142078d66bad842f0a69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.665994 13974 ts_tablet_manager.cc:1918] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:53.667389 13023 tablet_service.cc:1467] Processing CreateTablet for tablet 3127a5a12a9a433690dc5f8b0d474083 (DEFAULT_TABLE table=default.Table_829882defbbb45f78cd9bb86c3f7303f [id=4eebce3cc6e346fb92e0cd5b62af2c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.668773 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3127a5a12a9a433690dc5f8b0d474083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.669477 13974 log.cc:1196] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/081b4e1f53c74f768a4bf401ae22b585
I20260810 23:32:53.670395 13974 ts_tablet_manager.cc:1939] T 081b4e1f53c74f768a4bf401ae22b585 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:53.671326 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 206) took 4410 ms. Trace:
I20260810 23:32:53.671626 13974 rpcz_store.cc:276] 0810 23:32:49.260924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.264231 (+ 3307us) service_pool.cc:227] Handling call
0810 23:32:53.513889 (+4249658us) ts_tablet_manager.cc:1145] Deleting tablet 081b4e1f53c74f768a4bf401ae22b585
0810 23:32:53.513991 (+ 102us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.628387 (+114396us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.665675 (+ 37288us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.671265 (+ 5590us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11473,"tablet-delete.queue_time_us":4248876}
I20260810 23:32:53.673121 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 081b4e1f53c74f768a4bf401ae22b585 (table default.Table_fd94688e420b45c2a7f44c6b7db5d7e5 [id=cc72e0bbd749430abf4dabf1d737c92d]) successfully deleted
I20260810 23:32:53.673007 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 3127a5a12a9a433690dc5f8b0d474083 (DEFAULT_TABLE table=default.Table_829882defbbb45f78cd9bb86c3f7303f [id=4eebce3cc6e346fb92e0cd5b62af2c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.649019 12240 catalog_manager.cc:5495] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129). New cstate: current_term: 1 leader_uuid: "191d120f4c7f44fc8475305a6d2e75f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: HEALTHY } } }
I20260810 23:32:53.674239 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3127a5a12a9a433690dc5f8b0d474083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.674667 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:53.675518 13974 tablet_replica.cc:330] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:53.684610 13335 raft_consensus.cc:3037] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:53.685075 13974 raft_consensus.cc:2229] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:53.685642 13974 raft_consensus.cc:2258] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:53.687330 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 4166ff56ed074eca95dd909f36a8a28b (DEFAULT_TABLE table=default.Table_535019edc4634bb68e6c59f1e8c56bac [id=47f3c40c679f4fc0b16afcb03946d8a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.688575 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4166ff56ed074eca95dd909f36a8a28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.691640 13279 tablet_service.cc:1467] Processing CreateTablet for tablet 5d96bc27a87a41908f3ec838c228030d (DEFAULT_TABLE table=default.Table_3b3f1f30941b41d1a42bad9834394fe9 [id=af9ca26d147d47df98354d79ff9db2f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.693390 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d96bc27a87a41908f3ec838c228030d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.693643 13008 tablet_service.cc:1467] Processing CreateTablet for tablet 4166ff56ed074eca95dd909f36a8a28b (DEFAULT_TABLE table=default.Table_535019edc4634bb68e6c59f1e8c56bac [id=47f3c40c679f4fc0b16afcb03946d8a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.695688 13008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4166ff56ed074eca95dd909f36a8a28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.698433 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 4166ff56ed074eca95dd909f36a8a28b (DEFAULT_TABLE table=default.Table_535019edc4634bb68e6c59f1e8c56bac [id=47f3c40c679f4fc0b16afcb03946d8a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.701604 13974 ts_tablet_manager.cc:1905] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:53.703814 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 3127a5a12a9a433690dc5f8b0d474083 (DEFAULT_TABLE table=default.Table_829882defbbb45f78cd9bb86c3f7303f [id=4eebce3cc6e346fb92e0cd5b62af2c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.705478 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4166ff56ed074eca95dd909f36a8a28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.706115 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3127a5a12a9a433690dc5f8b0d474083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.708525 12228 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 303) took 1022 ms. Trace:
I20260810 23:32:53.708931 12228 rpcz_store.cc:276] 0810 23:32:52.686245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.688261 (+ 2016us) service_pool.cc:227] Handling call
0810 23:32:52.692272 (+ 4011us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:52.696715 (+ 4443us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:53.707639 (+1010924us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:53.708078 (+ 439us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:53.708432 (+ 354us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:53.544276 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:53.544778 (+ 502us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:53.544809 (+ 31us) write_op.cc:435] Acquired schema lock
0810 23:32:53.544825 (+ 16us) tablet.cc:660] Decoding operations
0810 23:32:53.545177 (+ 352us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:53.545227 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:53.545256 (+ 29us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:53.545512 (+ 256us) tablet.cc:699] Row locks acquired
0810 23:32:53.545531 (+ 19us) write_op.cc:260] PREPARE: finished
0810 23:32:53.545688 (+ 157us) write_op.cc:270] Start()
0810 23:32:53.545794 (+ 106us) write_op.cc:276] Timestamp: P: 1786404773545616 usec, L: 0
0810 23:32:53.545814 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:32:53.553661 (+ 7847us) log.cc:842] Serialized 433 byte log entry
0810 23:32:53.616104 (+ 62443us) op_driver.cc:444] REPLICATION: finished
0810 23:32:53.677683 (+ 61579us) write_op.cc:301] APPLY: starting
0810 23:32:53.677806 (+ 123us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:53.678047 (+ 241us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:53.678348 (+ 301us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:53.678990 (+ 642us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:53.679157 (+ 167us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:53.679203 (+ 46us) write_op.cc:312] APPLY: finished
0810 23:32:53.679467 (+ 264us) log.cc:842] Serialized 29 byte log entry
0810 23:32:53.680561 (+ 1094us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:53.680654 (+ 93us) write_op.cc:454] Released schema lock
0810 23:32:53.680695 (+ 41us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":61304,"num_ops":2,"prepare.queue_time_us":635180,"prepare.run_cpu_time_us":3021,"prepare.run_wall_time_us":51984,"raft.queue_time_us":18101,"raft.run_cpu_time_us":497,"raft.run_wall_time_us":497,"replication_time_us":70141,"threads_started":1}]]}
I20260810 23:32:53.650643 14801 raft_consensus.cc:504] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:53.715090 12952 consensus_peers.cc:487] T 3fef127263f1470ea1056a454971e465 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:53.716631 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdec13ee70d402fa3a08119e7bf1313" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:53.718365 12700 leader_election.cc:341] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:53.718688 14121 raft_consensus.cc:2229] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:53.720167 14801 leader_election.cc:288] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:53.724313 14121 raft_consensus.cc:2258] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:53.732614 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_d3429684c4a7448caafcc4052426b361"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:53.734987 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3429684c4a7448caafcc4052426b361 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:53.767853 12223 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32fad37a28e14903b8bde5aac6ee4e69 consensus state has no leader
I20260810 23:32:53.769546 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_e1ff8765e6e547fda12a138edb0a5bb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:53.772344 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1ff8765e6e547fda12a138edb0a5bb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:53.785321 14877 raft_consensus.cc:482] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f26517150f1c480a877f15aec71d020b)
I20260810 23:32:53.786185 14877 raft_consensus.cc:504] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:53.790598 14877 leader_election.cc:288] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:53.794827 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 5d96bc27a87a41908f3ec838c228030d (DEFAULT_TABLE table=default.Table_3b3f1f30941b41d1a42bad9834394fe9 [id=af9ca26d147d47df98354d79ff9db2f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.796442 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d96bc27a87a41908f3ec838c228030d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.628054 13935 raft_consensus.cc:2258] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:53.799329 13935 ts_tablet_manager.cc:1905] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:53.679404 13691 raft_consensus.cc:348] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:53.791666 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "423e8d583d704fa9834026ebcd57e17e" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:53.804514 14121 ts_tablet_manager.cc:1905] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:53.796327 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 5d96bc27a87a41908f3ec838c228030d (DEFAULT_TABLE table=default.Table_3b3f1f30941b41d1a42bad9834394fe9 [id=af9ca26d147d47df98354d79ff9db2f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.806043 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d96bc27a87a41908f3ec838c228030d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.810779 13007 tablet_service.cc:1467] Processing CreateTablet for tablet a391fd0c232549bb8b8bca740c9f866a (DEFAULT_TABLE table=default.Table_02819bd703c44810832aecd72d795582 [id=59b578941afe4d26a2fa5e09137579b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.812059 13007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a391fd0c232549bb8b8bca740c9f866a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.824180 13691 raft_consensus.cc:374] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:53.825105 13691 raft_consensus.cc:729] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:53.826335 13691 consensus_queue.cc:260] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:53.828898 13691 ts_tablet_manager.cc:1428] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.245s user 0.009s sys 0.000s
I20260810 23:32:53.845366 12755 tablet_service.cc:1467] Processing CreateTablet for tablet a391fd0c232549bb8b8bca740c9f866a (DEFAULT_TABLE table=default.Table_02819bd703c44810832aecd72d795582 [id=59b578941afe4d26a2fa5e09137579b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.881508 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a391fd0c232549bb8b8bca740c9f866a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.917739 14587 consensus_queue.cc:1035] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:53.934615 14792 consensus_queue.cc:1035] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:53.944164 13691 tablet_bootstrap.cc:492] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:54.008702 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 758c736a764048299360f27b0794d0d2 (DEFAULT_TABLE table=default.Table_b89242e803f549b8aca7b725f00cd775 [id=84fae17bae554c39a0c5135014b4ee36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:54.011885 13014 tablet_service.cc:1467] Processing CreateTablet for tablet fb8d482c6ee54e45a41201d98a5b60b4 (DEFAULT_TABLE table=default.Table_bf4e391ee20b45d393a269203b401bc0 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:53.800530 13279 tablet_service.cc:1467] Processing CreateTablet for tablet a391fd0c232549bb8b8bca740c9f866a (DEFAULT_TABLE table=default.Table_02819bd703c44810832aecd72d795582 [id=59b578941afe4d26a2fa5e09137579b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:54.016269 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a391fd0c232549bb8b8bca740c9f866a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.814174 13974 ts_tablet_manager.cc:1918] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260810 23:32:54.017808 13974 log.cc:1196] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/b261673faa964b65af3576c89e302864
I20260810 23:32:54.018775 13974 ts_tablet_manager.cc:1939] T b261673faa964b65af3576c89e302864 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:54.019872 13282 tablet_service.cc:1467] Processing CreateTablet for tablet fb8d482c6ee54e45a41201d98a5b60b4 (DEFAULT_TABLE table=default.Table_bf4e391ee20b45d393a269203b401bc0 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:54.021389 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8d482c6ee54e45a41201d98a5b60b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:53.817735 13665 tablet_bootstrap.cc:492] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:54.023738 13665 ts_tablet_manager.cc:1397] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.513s user 0.010s sys 0.007s
I20260810 23:32:53.885257 14842 raft_consensus.cc:482] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:54.024349 14842 raft_consensus.cc:504] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.024973 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 0df95e01ce894651a0f49396baf49c51 (DEFAULT_TABLE table=default.Table_00897e6950fb4847a8b9d4c8f709cc50 [id=25553ccfd84741069c668ac3c5be3d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:54.026227 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df95e01ce894651a0f49396baf49c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:54.031929 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 207) took 4770 ms. Trace:
I20260810 23:32:54.004240 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 758c736a764048299360f27b0794d0d2 (DEFAULT_TABLE table=default.Table_b89242e803f549b8aca7b725f00cd775 [id=84fae17bae554c39a0c5135014b4ee36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:54.046854 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 758c736a764048299360f27b0794d0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:54.046379 13974 rpcz_store.cc:276] 0810 23:32:49.261577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.265142 (+ 3565us) service_pool.cc:227] Handling call
0810 23:32:53.675332 (+4410190us) ts_tablet_manager.cc:1145] Deleting tablet b261673faa964b65af3576c89e302864
0810 23:32:53.675395 (+ 63us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:53.728725 (+ 53330us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:53.813832 (+ 85107us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:54.031844 (+218012us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4409259}
I20260810 23:32:53.879992 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdec13ee70d402fa3a08119e7bf1313" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:54.051568 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet b261673faa964b65af3576c89e302864 (table default.Table_adf76678e17a494185985a544def0f2d [id=5a377948adc7475c8c98141deaf1c274]) successfully deleted
I20260810 23:32:54.054766 13974 tablet_replica.cc:330] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:54.041736 14842 leader_election.cc:288] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:54.058779 13974 raft_consensus.cc:2229] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:54.035390 13665 raft_consensus.cc:348] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.059850 13665 raft_consensus.cc:374] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:54.060231 13665 raft_consensus.cc:729] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:54.061409 13691 tablet_bootstrap.cc:654] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:54.063834 14427 raft_consensus.cc:482] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:54.064410 14427 raft_consensus.cc:504] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.068598 13974 raft_consensus.cc:2258] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:54.069028 14587 raft_consensus.cc:482] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:54.069306 14427 leader_election.cc:288] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:54.069664 14587 raft_consensus.cc:504] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.070776 13974 ts_tablet_manager.cc:1905] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:54.070566 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5794637d5dd43f2b05edd53865f4e99" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:54.072912 12952 leader_election.cc:341] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:54.075493 14587 leader_election.cc:288] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:54.063314 13665 consensus_queue.cc:260] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:54.076974 12952 consensus_peers.cc:487] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:54.078125 13665 ts_tablet_manager.cc:1428] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20260810 23:32:54.078891 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d4c879ca6a402dbafe91940e7b367a" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:54.079972 13335 raft_consensus.cc:2454] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:54.081530 13665 tablet_bootstrap.cc:492] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:54.081908 12952 leader_election.cc:302] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:54.083117 14587 raft_consensus.cc:2781] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:54.083688 14587 raft_consensus.cc:482] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:54.084329 14587 raft_consensus.cc:3037] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:54.123503 12226 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ff43927ed204bfda6f647808b44fa3a consensus state has no leader
I20260810 23:32:54.126362 13278 tablet_service.cc:1467] Processing CreateTablet for tablet 136f5a8af8ae46cdb3dce5344fce59ad (DEFAULT_TABLE table=default.Table_cb78723192e84474b32e919ec39a002b [id=c99a6f0c231f49da9e354014bb4a3544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:54.129410 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136f5a8af8ae46cdb3dce5344fce59ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:54.165694 12764 tablet_service.cc:1467] Processing CreateTablet for tablet 758c736a764048299360f27b0794d0d2 (DEFAULT_TABLE table=default.Table_b89242e803f549b8aca7b725f00cd775 [id=84fae17bae554c39a0c5135014b4ee36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:54.167062 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 758c736a764048299360f27b0794d0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:54.172374 13514 tablet_bootstrap.cc:654] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:54.173602 12768 tablet_service.cc:1467] Processing CreateTablet for tablet fb8d482c6ee54e45a41201d98a5b60b4 (DEFAULT_TABLE table=default.Table_bf4e391ee20b45d393a269203b401bc0 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:54.174823 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8d482c6ee54e45a41201d98a5b60b4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:54.184900 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 84650.6us
I20260810 23:32:54.185626 12763 tablet_service.cc:1467] Processing CreateTablet for tablet 0df95e01ce894651a0f49396baf49c51 (DEFAULT_TABLE table=default.Table_00897e6950fb4847a8b9d4c8f709cc50 [id=25553ccfd84741069c668ac3c5be3d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:54.187543 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df95e01ce894651a0f49396baf49c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:54.209791 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 136f5a8af8ae46cdb3dce5344fce59ad (DEFAULT_TABLE table=default.Table_cb78723192e84474b32e919ec39a002b [id=c99a6f0c231f49da9e354014bb4a3544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:54.211457 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136f5a8af8ae46cdb3dce5344fce59ad. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:54.214257 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 209805us
I20260810 23:32:54.226388 13665 tablet_bootstrap.cc:654] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:54.240379 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5794637d5dd43f2b05edd53865f4e99" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:54.241814 12803 raft_consensus.cc:2454] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:54.244158 13023 tablet_service.cc:1467] Processing CreateTablet for tablet 0df95e01ce894651a0f49396baf49c51 (DEFAULT_TABLE table=default.Table_00897e6950fb4847a8b9d4c8f709cc50 [id=25553ccfd84741069c668ac3c5be3d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:54.244793 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d4c879ca6a402dbafe91940e7b367a" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:54.245761 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df95e01ce894651a0f49396baf49c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:54.253592 14792 raft_consensus.cc:482] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 191d120f4c7f44fc8475305a6d2e75f3)
W20260810 23:32:54.255079 12698 leader_election.cc:341] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:54.255714 12698 leader_election.cc:302] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:54.256714 14814 raft_consensus.cc:2726] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:54.257331 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 758c736a764048299360f27b0794d0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:54.279568 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0dd19112504f8ca57f64dcf0bd50a8" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:54.296981 14814 raft_consensus.cc:482] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:32:54.299013 13213 leader_election.cc:341] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 423e8d583d704fa9834026ebcd57e17e
I20260810 23:32:54.299516 13213 leader_election.cc:302] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:54.300607 14842 raft_consensus.cc:2726] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:32:54.313849 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8d482c6ee54e45a41201d98a5b60b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:54.316069 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 136f5a8af8ae46cdb3dce5344fce59ad (DEFAULT_TABLE table=default.Table_cb78723192e84474b32e919ec39a002b [id=c99a6f0c231f49da9e354014bb4a3544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:54.317593 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136f5a8af8ae46cdb3dce5344fce59ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:54.297878 14814 raft_consensus.cc:504] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:54.331912 14814 leader_election.cc:288] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:54.333698 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0dd19112504f8ca57f64dcf0bd50a8" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:54.335023 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0d9227d8574de8bd73041f3e8043a0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:54.337076 12698 leader_election.cc:341] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:54.337939 13974 ts_tablet_manager.cc:1918] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:54.343981 12224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 0df95e01ce894651a0f49396baf49c51
I20260810 23:32:54.345937 12224 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 297) took 2066 ms. Trace:
I20260810 23:32:54.346896 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet 0df95e01ce894651a0f49396baf49c51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:52 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:54.346294 12224 rpcz_store.cc:276] 0810 23:32:52.279349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.281761 (+ 2412us) service_pool.cc:227] Handling call
0810 23:32:52.283335 (+ 1574us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:52.358162 (+ 74827us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:52.358333 (+ 171us) catalog_manager.cc:2761] Locking tablets
0810 23:32:53.629293 (+1270960us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:54.343420 (+714127us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:54.343712 (+ 292us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:54.343848 (+ 136us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:54.345852 (+ 2004us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:54.131847 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:54.132315 (+ 468us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:54.132343 (+ 28us) write_op.cc:435] Acquired schema lock
0810 23:32:54.132358 (+ 15us) tablet.cc:660] Decoding operations
0810 23:32:54.133090 (+ 732us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:54.133146 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:54.133176 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:54.133643 (+ 467us) tablet.cc:699] Row locks acquired
0810 23:32:54.133659 (+ 16us) write_op.cc:260] PREPARE: finished
0810 23:32:54.133788 (+ 129us) write_op.cc:270] Start()
0810 23:32:54.133893 (+ 105us) write_op.cc:276] Timestamp: P: 1786404774132844 usec, L: 0
0810 23:32:54.133911 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 23:32:54.161297 (+ 27386us) log.cc:842] Serialized 734 byte log entry
0810 23:32:54.339254 (+177957us) op_driver.cc:444] REPLICATION: finished
0810 23:32:54.340382 (+ 1128us) write_op.cc:301] APPLY: starting
0810 23:32:54.340533 (+ 151us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:54.340662 (+ 129us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:54.340772 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:54.341074 (+ 302us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:54.341230 (+ 156us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:54.341354 (+ 124us) write_op.cc:312] APPLY: finished
0810 23:32:54.341715 (+ 361us) log.cc:842] Serialized 29 byte log entry
0810 23:32:54.342661 (+ 946us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:54.342745 (+ 84us) write_op.cc:454] Released schema lock
0810 23:32:54.342781 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":915,"apply.run_cpu_time_us":2043,"apply.run_wall_time_us":10810,"num_ops":2,"prepare.queue_time_us":500122,"prepare.run_cpu_time_us":3076,"prepare.run_wall_time_us":46732,"raft.queue_time_us":4731,"raft.run_cpu_time_us":982,"raft.run_wall_time_us":18232,"replication_time_us":205228}]]}
I20260810 23:32:54.359647 13974 log.cc:1196] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/133654008f5247b2ac87dcd09195f197
I20260810 23:32:54.359334 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_a7d468b7f476404c8b400f3923cd1543"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:54.349751 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet 0df95e01ce894651a0f49396baf49c51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:52 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:54.361795 13974 ts_tablet_manager.cc:1939] T 133654008f5247b2ac87dcd09195f197 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:54.362798 12237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet f0cba350e63e46a894a89ccc5727aab8
I20260810 23:32:54.364652 12237 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 302) took 1790 ms. Trace:
I20260810 23:32:54.369081 13007 tablet_service.cc:1514] Processing DeleteTablet for tablet 0df95e01ce894651a0f49396baf49c51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:52 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:54.369257 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet f0cba350e63e46a894a89ccc5727aab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:52 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:54.370464 13146 heartbeater.cc:347] Connected to a master server at 127.11.226.190:43533
I20260810 23:32:54.372179 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet f0cba350e63e46a894a89ccc5727aab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:52 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:54.373471 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet f0cba350e63e46a894a89ccc5727aab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:52 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:54.377135 13514 tablet_bootstrap.cc:492] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:54.377723 13514 ts_tablet_manager.cc:1397] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.814s user 0.023s sys 0.010s
I20260810 23:32:54.380942 13514 raft_consensus.cc:348] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.381840 13514 raft_consensus.cc:374] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:54.364966 12237 rpcz_store.cc:276] 0810 23:32:52.574810 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.574982 (+ 172us) service_pool.cc:227] Handling call
0810 23:32:52.641525 (+ 66543us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:52.642238 (+ 713us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:52.642515 (+ 277us) catalog_manager.cc:2761] Locking tablets
0810 23:32:53.644907 (+1002392us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:54.362336 (+717429us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:54.362594 (+ 258us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:54.362714 (+ 120us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:54.364582 (+ 1868us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:54.178755 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:54.179631 (+ 876us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:54.179661 (+ 30us) write_op.cc:435] Acquired schema lock
0810 23:32:54.179676 (+ 15us) tablet.cc:660] Decoding operations
0810 23:32:54.180148 (+ 472us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:54.180187 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:54.180222 (+ 35us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:54.180599 (+ 377us) tablet.cc:699] Row locks acquired
0810 23:32:54.180614 (+ 15us) write_op.cc:260] PREPARE: finished
0810 23:32:54.180770 (+ 156us) write_op.cc:270] Start()
0810 23:32:54.180888 (+ 118us) write_op.cc:276] Timestamp: P: 1786404774179817 usec, L: 0
0810 23:32:54.180905 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:32:54.181649 (+ 744us) log.cc:842] Serialized 734 byte log entry
0810 23:32:54.339530 (+157881us) op_driver.cc:444] REPLICATION: finished
0810 23:32:54.346332 (+ 6802us) write_op.cc:301] APPLY: starting
0810 23:32:54.346435 (+ 103us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:54.346609 (+ 174us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:54.346734 (+ 125us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:54.350637 (+ 3903us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:54.350867 (+ 230us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:54.350954 (+ 87us) write_op.cc:312] APPLY: finished
0810 23:32:54.351302 (+ 348us) log.cc:842] Serialized 29 byte log entry
0810 23:32:54.351541 (+ 239us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:54.351619 (+ 78us) write_op.cc:454] Released schema lock
0810 23:32:54.351654 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6550,"num_ops":2,"prepare.queue_time_us":530820,"raft.queue_time_us":127,"replication_time_us":158540,"thread_start_us":9650,"threads_started":2}]]}
I20260810 23:32:54.383486 12237 master_service.cc:946] ReplaceTablet: received request to replace tablet 806fe5162aef4d36ad5e2ac324047426 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:54.386659 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_f07540d9f70a42a99acf3b3613c98d18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:54.389282 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f07540d9f70a42a99acf3b3613c98d18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:54.387880 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 209) took 5107 ms. Trace:
I20260810 23:32:54.390269 13974 rpcz_store.cc:276] 0810 23:32:49.280514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.280698 (+ 184us) service_pool.cc:227] Handling call
0810 23:32:54.054558 (+4773860us) ts_tablet_manager.cc:1145] Deleting tablet 133654008f5247b2ac87dcd09195f197
0810 23:32:54.054628 (+ 70us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:54.235362 (+180734us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:54.337625 (+102263us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:54.387779 (+ 50154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4772947}
I20260810 23:32:54.394160 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 133654008f5247b2ac87dcd09195f197 (table default.Table_d664d07d116e4f2aa4fc263449eb8dd0 [id=04ed6428e4df49b2ab0438035da70731]) successfully deleted
I20260810 23:32:54.395638 13974 tablet_replica.cc:330] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:54.397575 13974 raft_consensus.cc:2229] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
W20260810 23:32:54.401508 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7d468b7f476404c8b400f3923cd1543 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:54.382154 13514 raft_consensus.cc:729] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:54.416381 13974 raft_consensus.cc:2258] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:54.400539 13665 tablet_bootstrap.cc:492] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:54.418310 13665 ts_tablet_manager.cc:1397] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.337s user 0.020s sys 0.008s
I20260810 23:32:54.422569 14877 raft_consensus.cc:482] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:54.421950 13665 raft_consensus.cc:348] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.423472 13665 raft_consensus.cc:374] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:54.423722 13665 raft_consensus.cc:729] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:54.423250 14877 raft_consensus.cc:504] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.426155 14877 leader_election.cc:288] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:54.418111 13514 consensus_queue.cc:260] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.427987 13974 ts_tablet_manager.cc:1905] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:54.428383 13665 consensus_queue.cc:260] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.440898 14814 raft_consensus.cc:482] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:54.441401 14814 raft_consensus.cc:504] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.442059 13665 ts_tablet_manager.cc:1428] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.023s user 0.004s sys 0.002s
I20260810 23:32:54.443571 14814 leader_election.cc:288] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:54.449028 13514 ts_tablet_manager.cc:1428] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.071s user 0.011s sys 0.000s
I20260810 23:32:54.448722 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10033af79e854103bc47a046d2219ca0" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:54.449952 12803 raft_consensus.cc:2379] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in current term 1: Already voted for candidate f26517150f1c480a877f15aec71d020b in this term.
I20260810 23:32:54.449725 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bce222cd5ab41fdb12fff3fe3dfbd3c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:54.451447 12698 leader_election.cc:341] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:54.455654 13665 tablet_bootstrap.cc:492] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:54.456503 14814 raft_consensus.cc:482] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:54.457080 14814 raft_consensus.cc:504] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.461402 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d1b15dd545c4598aa5c7a9dc53618ec" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:54.462908 12698 leader_election.cc:341] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:54.464395 14814 leader_election.cc:288] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:54.478413 13514 tablet_bootstrap.cc:492] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:54.479653 13665 tablet_bootstrap.cc:654] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:54.504436 13665 tablet_bootstrap.cc:492] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:54.504973 13665 ts_tablet_manager.cc:1397] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.012s
I20260810 23:32:54.294981 14792 raft_consensus.cc:504] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.523115 13665 raft_consensus.cc:348] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.525030 13665 raft_consensus.cc:374] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:54.525277 13665 raft_consensus.cc:729] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:54.527247 13665 consensus_queue.cc:260] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.533349 13514 tablet_bootstrap.cc:654] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:54.541476 13665 ts_tablet_manager.cc:1428] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.036s user 0.008s sys 0.000s
W20260810 23:32:54.545532 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 0ff43927ed204bfda6f647808b44fa3a (table default.Table_dc4c377dd45047ba9e9c97d06a255575 [id=e539d34e2b7c48bebe9e2729946b3856], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0ff43927ed204bfda6f647808b44fa3a consensus state has no leader
I20260810 23:32:54.549338 13665 tablet_bootstrap.cc:492] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:54.556793 13974 ts_tablet_manager.cc:1918] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:54.557230 13974 log.cc:1196] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/13ad5d63a2104995990a86a000963cdc
I20260810 23:32:54.558000 13974 ts_tablet_manager.cc:1939] T 13ad5d63a2104995990a86a000963cdc P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:54.572808 13665 tablet_bootstrap.cc:654] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:54.581475 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 210) took 5269 ms. Trace:
I20260810 23:32:54.582036 13974 rpcz_store.cc:276] 0810 23:32:49.311697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.312597 (+ 900us) service_pool.cc:227] Handling call
0810 23:32:54.395384 (+5082787us) ts_tablet_manager.cc:1145] Deleting tablet 13ad5d63a2104995990a86a000963cdc
0810 23:32:54.395496 (+ 112us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:54.501627 (+106131us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:54.556496 (+ 54869us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:54.581380 (+ 24884us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5081959}
I20260810 23:32:54.583751 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 13ad5d63a2104995990a86a000963cdc (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5]) successfully deleted
I20260810 23:32:54.592734 13974 tablet_replica.cc:330] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:54.594017 13974 raft_consensus.cc:2229] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:54.595456 13974 raft_consensus.cc:2258] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:54.594511 13514 tablet_bootstrap.cc:492] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:54.597280 13514 ts_tablet_manager.cc:1397] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.119s user 0.016s sys 0.012s
I20260810 23:32:54.598490 13974 ts_tablet_manager.cc:1905] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:54.600051 13665 tablet_bootstrap.cc:492] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:54.600623 13665 ts_tablet_manager.cc:1397] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.052s user 0.021s sys 0.000s
I20260810 23:32:54.604332 13665 raft_consensus.cc:348] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:54.612586 13416 tablet.cc:2369] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:54.616211 13665 raft_consensus.cc:374] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:54.616582 13665 raft_consensus.cc:729] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:54.617338 13665 consensus_queue.cc:260] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.634848 13665 ts_tablet_manager.cc:1428] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20260810 23:32:54.640322 13665 tablet_bootstrap.cc:492] T e6b8bf38da5647889756935404caad75 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:54.643273 14877 raft_consensus.cc:482] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:54.643925 14877 raft_consensus.cc:504] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.647234 14877 leader_election.cc:288] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
W20260810 23:32:54.650027 13213 leader_election.cc:341] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 440de7d9de8d4be9a5322e1fe3388b9b
I20260810 23:32:54.652261 14121 ts_tablet_manager.cc:1918] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:54.652725 14121 log.cc:1196] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/081b4e1f53c74f768a4bf401ae22b585
I20260810 23:32:54.653555 14121 ts_tablet_manager.cc:1939] T 081b4e1f53c74f768a4bf401ae22b585 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:54.654615 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 198) took 5340 ms. Trace:
I20260810 23:32:54.654893 14121 rpcz_store.cc:276] 0810 23:32:49.313739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.318472 (+ 4733us) service_pool.cc:227] Handling call
0810 23:32:53.618103 (+4299631us) ts_tablet_manager.cc:1145] Deleting tablet 081b4e1f53c74f768a4bf401ae22b585
0810 23:32:53.618158 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:54.170344 (+552186us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:54.651913 (+481569us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:54.654554 (+ 2641us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12856,"tablet-delete.queue_time_us":4299025}
I20260810 23:32:54.656231 14121 tablet_replica.cc:330] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:54.661026 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db363eb0f98d460eb6f5a577233d8ac0" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:54.661779 13338 raft_consensus.cc:2454] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
W20260810 23:32:54.663199 13008 consensus_meta.cc:287] T 4166ff56ed074eca95dd909f36a8a28b P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.710s user 0.008s sys 0.012s
W20260810 23:32:54.669734 14587 consensus_meta.cc:287] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.585s user 0.011s sys 0.008s
I20260810 23:32:54.670406 14587 raft_consensus.cc:504] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.677021 14792 leader_election.cc:288] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:54.683054 13665 tablet_bootstrap.cc:654] T e6b8bf38da5647889756935404caad75 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:54.682992 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 081b4e1f53c74f768a4bf401ae22b585 (table default.Table_fd94688e420b45c2a7f44c6b7db5d7e5 [id=cc72e0bbd749430abf4dabf1d737c92d]) successfully deleted
I20260810 23:32:54.690691 13514 raft_consensus.cc:348] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.691718 13514 raft_consensus.cc:374] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:54.691991 13514 raft_consensus.cc:729] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:54.692806 13514 consensus_queue.cc:260] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.696332 14121 raft_consensus.cc:2229] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:54.705286 13974 ts_tablet_manager.cc:1918] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:54.705850 13974 log.cc:1196] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/42517f11422b42c39db282e2c5f584a3
I20260810 23:32:54.706727 13974 ts_tablet_manager.cc:1939] T 42517f11422b42c39db282e2c5f584a3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:54.717397 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 211) took 5289 ms. Trace:
I20260810 23:32:54.717923 13974 rpcz_store.cc:276] 0810 23:32:49.427664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.436743 (+ 9079us) service_pool.cc:227] Handling call
0810 23:32:54.592378 (+5155635us) ts_tablet_manager.cc:1145] Deleting tablet 42517f11422b42c39db282e2c5f584a3
0810 23:32:54.592535 (+ 157us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:54.675391 (+ 82856us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:54.703419 (+ 28028us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:54.717284 (+ 13865us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5153329}
I20260810 23:32:54.719774 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 42517f11422b42c39db282e2c5f584a3 (table default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f]) successfully deleted
I20260810 23:32:54.720932 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 212) took 5075 ms. Trace:
I20260810 23:32:54.721333 13974 rpcz_store.cc:276] 0810 23:32:49.645998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.646155 (+ 157us) service_pool.cc:227] Handling call
0810 23:32:54.720217 (+5074062us) ts_tablet_manager.cc:1145] Deleting tablet 240adfa4be844518bbc9624d0e408789
0810 23:32:54.720332 (+ 115us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:54.720856 (+ 524us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5073211}
I20260810 23:32:54.725455 13974 tablet_replica.cc:330] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:54.727638 13665 tablet_bootstrap.cc:492] T e6b8bf38da5647889756935404caad75 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:54.728317 13665 ts_tablet_manager.cc:1397] T e6b8bf38da5647889756935404caad75 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.012s
I20260810 23:32:54.729499 13974 raft_consensus.cc:2229] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:54.731281 13974 raft_consensus.cc:2258] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:54.732970 13514 ts_tablet_manager.cc:1428] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.135s user 0.008s sys 0.002s
I20260810 23:32:54.734154 13974 ts_tablet_manager.cc:1905] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:54.738458 14877 raft_consensus.cc:482] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:54.739113 14877 raft_consensus.cc:504] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.741183 13514 tablet_bootstrap.cc:492] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:54.744596 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b81e7056a441d2854ba5ab6b4c5433" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:54.745362 12803 raft_consensus.cc:2454] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
W20260810 23:32:54.745353 12218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.188:45477, user_credentials={real_user=slave}} blocked reactor thread for 142032us
I20260810 23:32:54.747483 13213 leader_election.cc:302] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:54.748515 14842 raft_consensus.cc:2781] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:54.746861 13665 raft_consensus.cc:348] T e6b8bf38da5647889756935404caad75 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.748870 14842 raft_consensus.cc:482] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:54.748996 13665 raft_consensus.cc:374] T e6b8bf38da5647889756935404caad75 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:54.749217 14842 raft_consensus.cc:3037] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:54.749250 13665 raft_consensus.cc:729] T e6b8bf38da5647889756935404caad75 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:54.750896 13665 consensus_queue.cc:260] T e6b8bf38da5647889756935404caad75 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.751561 13935 ts_tablet_manager.cc:1918] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:54.752737 14877 leader_election.cc:288] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:54.760769 14842 raft_consensus.cc:504] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.770042 14121 raft_consensus.cc:2258] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:54.772665 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 127951us
I20260810 23:32:54.777503 13665 ts_tablet_manager.cc:1428] T e6b8bf38da5647889756935404caad75 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.049s user 0.007s sys 0.003s
I20260810 23:32:54.790342 13665 tablet_bootstrap.cc:492] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:54.791710 14587 leader_election.cc:288] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:54.804275 13665 tablet_bootstrap.cc:654] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:54.751989 13935 log.cc:1196] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/c3d4c879ca6a402dbafe91940e7b367a
I20260810 23:32:54.805697 13935 ts_tablet_manager.cc:1939] T c3d4c879ca6a402dbafe91940e7b367a P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:54.807163 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 236) took 2722 ms. Trace:
I20260810 23:32:54.808302 12952 leader_election.cc:302] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:54.807662 13935 rpcz_store.cc:276] 0810 23:32:52.084610 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.084788 (+ 178us) service_pool.cc:227] Handling call
0810 23:32:53.626170 (+1541382us) ts_tablet_manager.cc:1145] Deleting tablet c3d4c879ca6a402dbafe91940e7b367a
0810 23:32:53.626230 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:54.175938 (+549708us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:54.751196 (+575258us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:54.807074 (+ 55878us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1540688}
I20260810 23:32:54.811554 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet c3d4c879ca6a402dbafe91940e7b367a (table default.Table_6e13e5c0a9da4cc5a1b85b4a247f8a3f [id=820d73489eed49d8a72357b3ac31300d]) successfully deleted
I20260810 23:32:54.815909 13514 tablet_bootstrap.cc:654] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:54.818951 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b81e7056a441d2854ba5ab6b4c5433" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:54.819664 12803 raft_consensus.cc:3037] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:54.823897 13665 tablet_bootstrap.cc:492] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:54.824487 13665 ts_tablet_manager.cc:1397] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.002s
I20260810 23:32:54.827512 13665 raft_consensus.cc:348] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:54.828375 13665 raft_consensus.cc:374] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:54.828681 13665 raft_consensus.cc:729] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:54.831660 13514 tablet_bootstrap.cc:492] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:54.832326 13514 ts_tablet_manager.cc:1397] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.008s
I20260810 23:32:54.832753 13665 consensus_queue.cc:260] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:54.834489 12803 raft_consensus.cc:2454] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:54.833488 14842 leader_election.cc:288] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:54.837905 13213 leader_election.cc:302] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:54.839072 12235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 4f453bd72168429086837b2b9b09fc78
I20260810 23:32:54.844679 14928 raft_consensus.cc:2781] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:54.845224 14928 raft_consensus.cc:686] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:54.835083 13514 raft_consensus.cc:348] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.845737 13514 raft_consensus.cc:374] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:54.846022 13514 raft_consensus.cc:729] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:54.846127 14928 consensus_queue.cc:237] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:54.847882 12891 tablet.cc:2369] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:54.848963 12235 catalog_manager.cc:6565] ReplaceTablet: tablet 4f453bd72168429086837b2b9b09fc78 deleted and replaced by tablet 09bdc7c2aace45da9fce81c4904db5d4
I20260810 23:32:54.849337 12235 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 305) took 1891 ms. Trace:
I20260810 23:32:54.852499 13278 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f453bd72168429086837b2b9b09fc78 with delete_type TABLET_DATA_DELETED (replaced by tablet 09bdc7c2aace45da9fce81c4904db5d4) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:54.853948 12769 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f453bd72168429086837b2b9b09fc78 with delete_type TABLET_DATA_DELETED (replaced by tablet 09bdc7c2aace45da9fce81c4904db5d4) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:54.853878 14877 raft_consensus.cc:482] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f26517150f1c480a877f15aec71d020b)
I20260810 23:32:54.854454 14877 raft_consensus.cc:504] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:54.856172 13514 consensus_queue.cc:260] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.856750 14877 leader_election.cc:288] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:54.857095 14814 raft_consensus.cc:482] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:54.857626 14814 raft_consensus.cc:504] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:54.860468 13213 leader_election.cc:341] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 4d697e7533af4fc0a15cb41f3fcce4db
I20260810 23:32:54.864152 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9443fb32d79941d58b13607e8ac064cd" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:54.864785 13514 ts_tablet_manager.cc:1428] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.032s user 0.008s sys 0.001s
W20260810 23:32:54.866156 12698 leader_election.cc:341] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:54.870172 14814 leader_election.cc:288] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:54.870682 13514 tablet_bootstrap.cc:492] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:54.849639 12235 rpcz_store.cc:276] 0810 23:32:52.958293 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.958576 (+ 283us) service_pool.cc:227] Handling call
0810 23:32:54.849275 (+1890699us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:54.400420 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:54.408853 (+ 8433us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:54.408881 (+ 28us) write_op.cc:435] Acquired schema lock
0810 23:32:54.408895 (+ 14us) tablet.cc:660] Decoding operations
0810 23:32:54.409244 (+ 349us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:54.409288 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:54.409309 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:54.409518 (+ 209us) tablet.cc:699] Row locks acquired
0810 23:32:54.409531 (+ 13us) write_op.cc:260] PREPARE: finished
0810 23:32:54.409656 (+ 125us) write_op.cc:270] Start()
0810 23:32:54.409744 (+ 88us) write_op.cc:276] Timestamp: P: 1786404774409616 usec, L: 0
0810 23:32:54.409760 (+ 16us) op_driver.cc:331] REPLICATION: starting
0810 23:32:54.575570 (+165810us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b5400032198. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce803f0c4 00007f0ce72eb4f9 00007f0ce72a6f91 00007f0ce72a6c6a 00007f0ce7312cd5 00007f0ce7313179 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9
0810 23:32:54.576069 (+ 499us) log.cc:842] Serialized 631 byte log entry
0810 23:32:54.795612 (+219543us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b58000839b0. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce7291834 00007f0ce7296771 00007f0ce72966f9 00007f0ce7296689 00007f0ce7296651 00007f0ce729581d 0000000000580784 00000000005805b9 00007f0ce03c26b6 00007f0ce03c5b11
0810 23:32:54.834669 (+ 39057us) op_driver.cc:444] REPLICATION: finished
0810 23:32:54.835667 (+ 998us) write_op.cc:301] APPLY: starting
0810 23:32:54.835763 (+ 96us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:54.835921 (+ 158us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:54.835943 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:54.836568 (+ 625us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:54.836843 (+ 275us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:54.836912 (+ 69us) write_op.cc:312] APPLY: finished
0810 23:32:54.837213 (+ 301us) log.cc:842] Serialized 29 byte log entry
0810 23:32:54.837429 (+ 216us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:54.837805 (+ 376us) write_op.cc:454] Released schema lock
0810 23:32:54.837861 (+ 56us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:54.838119 (+ 258us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:54.838143 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":759,"apply.run_cpu_time_us":2717,"apply.run_wall_time_us":2716,"num_ops":2,"prepare.queue_time_us":737443,"prepare.run_cpu_time_us":5453,"prepare.run_wall_time_us":175997,"raft.queue_time_us":6236,"raft.run_cpu_time_us":2821,"raft.run_wall_time_us":213059,"replication_time_us":424799,"spinlock_wait_cycles":764735744}]]}
I20260810 23:32:54.877803 13665 ts_tablet_manager.cc:1428] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.053s user 0.007s sys 0.001s
I20260810 23:32:54.879220 13665 tablet_bootstrap.cc:492] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:54.884641 13514 tablet_bootstrap.cc:654] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:54.905318 13665 tablet_bootstrap.cc:654] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:54.932523 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 124319us
I20260810 23:32:54.935808 13514 tablet_bootstrap.cc:492] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:54.936455 13514 ts_tablet_manager.cc:1397] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.066s user 0.018s sys 0.002s
I20260810 23:32:54.939881 13665 tablet_bootstrap.cc:492] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:54.940531 13665 ts_tablet_manager.cc:1397] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.062s user 0.013s sys 0.010s
I20260810 23:32:54.940601 13514 raft_consensus.cc:348] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.941581 13514 raft_consensus.cc:374] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:54.941897 13514 raft_consensus.cc:729] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:54.942713 13514 consensus_queue.cc:260] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:54.952214 12952 consensus_peers.cc:487] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 133654008f5247b2ac87dcd09195f197. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:54.953114 13665 raft_consensus.cc:348] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.953922 13665 raft_consensus.cc:374] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:54.954165 13665 raft_consensus.cc:729] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:54.954886 13665 consensus_queue.cc:260] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.957350 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d4c879ca6a402dbafe91940e7b367a" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:54.958117 13338 raft_consensus.cc:3037] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:54.959376 13665 ts_tablet_manager.cc:1428] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260810 23:32:54.963171 13665 tablet_bootstrap.cc:492] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:54.965590 13514 ts_tablet_manager.cc:1428] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.029s user 0.008s sys 0.000s
I20260810 23:32:54.966630 14814 raft_consensus.cc:482] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:54.967271 14814 raft_consensus.cc:504] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:54.971982 13514 tablet_bootstrap.cc:492] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:54.971679 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e8311c941b43058384cee6d8263dc1" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:54.973572 12698 leader_election.cc:341] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:54.970858 14814 leader_election.cc:288] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:54.982385 13974 ts_tablet_manager.cc:1918] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260810 23:32:54.982872 13974 log.cc:1196] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/08c45b71432a45269cd6a106fc5f7292
I20260810 23:32:54.983675 13665 tablet_bootstrap.cc:654] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:54.992328 14928 raft_consensus.cc:482] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:54.993224 14928 raft_consensus.cc:504] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.001550 13514 tablet_bootstrap.cc:654] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:55.001888 14928 leader_election.cc:288] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:55.003681 13691 tablet_bootstrap.cc:492] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:55.004262 13691 ts_tablet_manager.cc:1397] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 1.122s user 0.013s sys 0.022s
I20260810 23:32:55.003638 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fef127263f1470ea1056a454971e465" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:54.983871 14792 raft_consensus.cc:2781] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:32:55.011601 13008 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f453bd72168429086837b2b9b09fc78 with delete_type TABLET_DATA_DELETED (replaced by tablet 09bdc7c2aace45da9fce81c4904db5d4) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:55.014249 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 234) took 2933 ms. Trace:
I20260810 23:32:55.020637 13691 raft_consensus.cc:348] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.021716 13691 raft_consensus.cc:374] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:55.021972 13691 raft_consensus.cc:729] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:55.023957 13336 raft_consensus.cc:3037] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:55.026783 13327 raft_consensus.cc:3037] T e6b8bf38da5647889756935404caad75 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:55.031401 14121 ts_tablet_manager.cc:1905] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:55.034840 13665 tablet_bootstrap.cc:492] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:55.035372 13665 ts_tablet_manager.cc:1397] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.072s user 0.015s sys 0.000s
I20260810 23:32:55.011070 14792 raft_consensus.cc:482] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 191d120f4c7f44fc8475305a6d2e75f3)
I20260810 23:32:55.039034 14792 raft_consensus.cc:3037] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:32:55.039398 13665 raft_consensus.cc:348] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.040354 13665 raft_consensus.cc:374] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:55.040633 13665 raft_consensus.cc:729] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:55.035852 13514 tablet_bootstrap.cc:492] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:55.041788 13514 ts_tablet_manager.cc:1397] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.070s user 0.018s sys 0.000s
I20260810 23:32:55.041424 13665 consensus_queue.cc:260] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.047716 13665 ts_tablet_manager.cc:1428] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20260810 23:32:55.051147 13665 tablet_bootstrap.cc:492] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:55.022574 13691 consensus_queue.cc:260] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.058301 13514 raft_consensus.cc:348] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:55.061714 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 288225us
I20260810 23:32:55.059377 13514 raft_consensus.cc:374] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:55.072397 13974 ts_tablet_manager.cc:1939] T 08c45b71432a45269cd6a106fc5f7292 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:55.071641 13514 raft_consensus.cc:729] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:55.074651 13514 consensus_queue.cc:260] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.014951 13935 rpcz_store.cc:276] 0810 23:32:52.075887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.086690 (+ 10803us) service_pool.cc:227] Handling call
0810 23:32:55.008530 (+2921840us) ts_tablet_manager.cc:1145] Deleting tablet 13ad5d63a2104995990a86a000963cdc
0810 23:32:55.008604 (+ 74us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.009176 (+ 572us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2920926}
I20260810 23:32:55.083021 13935 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 237) took 2710 ms. Trace:
I20260810 23:32:55.085404 13691 ts_tablet_manager.cc:1428] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.081s user 0.008s sys 0.000s
I20260810 23:32:55.089048 13691 tablet_bootstrap.cc:492] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:55.097020 14792 raft_consensus.cc:504] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.100343 14908 consensus_queue.cc:1035] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:55.103507 12958 leader_election.cc:302] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:55.083562 13935 rpcz_store.cc:276] 0810 23:32:52.373003 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.373170 (+ 167us) service_pool.cc:227] Handling call
0810 23:32:55.082394 (+2709224us) ts_tablet_manager.cc:1145] Deleting tablet d603569c54ef4ffba9941d0cc3e0f97f
0810 23:32:55.082470 (+ 76us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.082922 (+ 452us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2645263}
I20260810 23:32:55.112747 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 213) took 5466 ms. Trace:
I20260810 23:32:55.113767 13974 rpcz_store.cc:276] 0810 23:32:49.646503 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.646638 (+ 135us) service_pool.cc:227] Handling call
0810 23:32:54.725168 (+5078530us) ts_tablet_manager.cc:1145] Deleting tablet 08c45b71432a45269cd6a106fc5f7292
0810 23:32:54.725283 (+ 115us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:54.898139 (+172856us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:54.982046 (+ 83907us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:55.112660 (+130614us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5077940}
I20260810 23:32:55.113520 14908 raft_consensus.cc:2781] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:55.116061 14908 raft_consensus.cc:482] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:55.117137 14908 raft_consensus.cc:3037] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:55.113636 12958 leader_election.cc:341] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: Tablet not RUNNING: STOPPED
I20260810 23:32:55.100821 13014 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 234) took 1093 ms. Trace:
I20260810 23:32:55.126417 13514 ts_tablet_manager.cc:1428] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.084s user 0.011s sys 0.000s
I20260810 23:32:55.128547 13514 tablet_bootstrap.cc:492] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:55.138471 13665 tablet_bootstrap.cc:654] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:55.139874 13974 tablet_replica.cc:330] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:55.147011 13974 raft_consensus.cc:2229] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:55.147703 13974 raft_consensus.cc:2258] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:55.154286 13974 ts_tablet_manager.cc:1905] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:55.161170 13338 raft_consensus.cc:2454] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:55.164233 12952 leader_election.cc:302] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:55.165808 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db363eb0f98d460eb6f5a577233d8ac0" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:55.166525 13334 raft_consensus.cc:3037] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 2
I20260810 23:32:55.183764 14792 leader_election.cc:288] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:55.186851 13691 tablet_bootstrap.cc:654] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:55.125042 13014 rpcz_store.cc:276] 0810 23:32:54.007646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:54.010932 (+ 3286us) service_pool.cc:227] Handling call
0810 23:32:54.251784 (+240852us) spinlock_profiling.cc:243] Waited 83.4 ms on lock 0x7b5800001898. stack: 00007fa86ed56033 00007fa86ed55f30 00007fa86e8d7534 00000000004db2b4 00000000004db249 00007fa86ebccf11 00007fa86ed50851 00007fa86ed50702 00007fa877be7b09 00007fa876c44730 00007fa876d26d21 00007fa876d041e3 00007fa876cb5869 00007fa8753a7c44 00007fa8753a7bd2 00007fa8753a7b01
0810 23:32:54.259030 (+ 7246us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:54.302594 (+ 43564us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:54.666983 (+364389us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:55.096859 (+429876us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:55.100666 (+ 3807us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":191852288}
I20260810 23:32:55.200872 14931 raft_consensus.cc:2781] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:55.201604 14931 raft_consensus.cc:686] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:55.202708 14931 consensus_queue.cc:237] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.207849 13334 raft_consensus.cc:2454] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 2.
I20260810 23:32:55.221393 13514 tablet_bootstrap.cc:654] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:55.225972 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0d9227d8574de8bd73041f3e8043a0" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:55.219229 14121 ts_tablet_manager.cc:1918] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260810 23:32:55.231177 14121 log.cc:1196] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/08c45b71432a45269cd6a106fc5f7292
I20260810 23:32:55.232183 14121 ts_tablet_manager.cc:1939] T 08c45b71432a45269cd6a106fc5f7292 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:55.239084 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bce222cd5ab41fdb12fff3fe3dfbd3c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:55.245476 12952 leader_election.cc:302] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:55.249495 13665 tablet_bootstrap.cc:492] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:55.253352 13665 ts_tablet_manager.cc:1397] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.202s user 0.012s sys 0.013s
I20260810 23:32:55.254050 14920 raft_consensus.cc:2781] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:32:55.254576 14920 raft_consensus.cc:686] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 2 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:55.255645 14920 consensus_queue.cc:237] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.262645 13665 raft_consensus.cc:348] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.263527 13665 raft_consensus.cc:374] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:55.270270 13514 tablet_bootstrap.cc:492] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:55.270184 13334 raft_consensus.cc:3037] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:55.270835 13514 ts_tablet_manager.cc:1397] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.143s user 0.018s sys 0.000s
I20260810 23:32:55.285982 13514 raft_consensus.cc:348] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.287156 13514 raft_consensus.cc:374] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:55.287477 13514 raft_consensus.cc:729] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:55.286898 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_75ca982c15e34c8d9603f100dff5b392 [id=b809029bf6cc48ab917e48bbf2109ae1]) successfully deleted
I20260810 23:32:55.290591 13691 tablet_bootstrap.cc:492] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:55.291268 13691 ts_tablet_manager.cc:1397] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.202s user 0.014s sys 0.001s
W20260810 23:32:55.304342 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 13ad5d63a2104995990a86a000963cdc because the tablet was not found. No further retry: Not found: Tablet not found: 13ad5d63a2104995990a86a000963cdc
W20260810 23:32:55.306792 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet d603569c54ef4ffba9941d0cc3e0f97f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d603569c54ef4ffba9941d0cc3e0f97f already in progress: creating tablet
W20260810 23:32:55.308419 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 0df95e01ce894651a0f49396baf49c51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0df95e01ce894651a0f49396baf49c51 already in progress: creating tablet
W20260810 23:32:55.309849 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet f0cba350e63e46a894a89ccc5727aab8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0cba350e63e46a894a89ccc5727aab8 already in progress: creating tablet
W20260810 23:32:55.311311 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 4f453bd72168429086837b2b9b09fc78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f453bd72168429086837b2b9b09fc78 already in progress: creating tablet
I20260810 23:32:55.312389 14928 raft_consensus.cc:482] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:32:55.315168 13151 tablet.cc:2369] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:55.263830 13665 raft_consensus.cc:729] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:55.318246 13665 consensus_queue.cc:260] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.328500 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9443fb32d79941d58b13607e8ac064cd" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:55.332216 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e8311c941b43058384cee6d8263dc1" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:55.337630 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d1b15dd545c4598aa5c7a9dc53618ec" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:55.342202 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 200) took 5759 ms. Trace:
I20260810 23:32:55.342778 14121 rpcz_store.cc:276] 0810 23:32:49.582617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.582754 (+ 137us) service_pool.cc:227] Handling call
0810 23:32:54.656042 (+5073288us) ts_tablet_manager.cc:1145] Deleting tablet 08c45b71432a45269cd6a106fc5f7292
0810 23:32:54.656095 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.083446 (+427351us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:55.218927 (+135481us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:55.342120 (+123193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39204,"tablet-delete.queue_time_us":5072133}
I20260810 23:32:55.344573 14928 raft_consensus.cc:504] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.347856 14928 leader_election.cc:288] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:55.350927 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10033af79e854103bc47a046d2219ca0" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:55.356177 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b81e7056a441d2854ba5ab6b4c5433" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:55.358521 13665 ts_tablet_manager.cc:1428] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.105s user 0.007s sys 0.001s
I20260810 23:32:55.369174 13691 raft_consensus.cc:348] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.383283 14974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e964330cf113403a8aa5b73ddd3b31d0)
I20260810 23:32:55.393430 14974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:55.359164 13974 ts_tablet_manager.cc:1918] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:55.360328 13665 tablet_bootstrap.cc:492] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:55.399955 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 08c45b71432a45269cd6a106fc5f7292 (table default.Table_75ca982c15e34c8d9603f100dff5b392 [id=b809029bf6cc48ab917e48bbf2109ae1]) successfully deleted
I20260810 23:32:55.381950 13212 leader_election.cc:302] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
W20260810 23:32:55.400655 12215 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 56247.4us
I20260810 23:32:55.402009 14877 raft_consensus.cc:2726] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:32:55.365204 14928 raft_consensus.cc:482] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:55.403671 14928 raft_consensus.cc:504] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:55.408658 12700 leader_election.cc:341] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:55.409484 12700 leader_election.cc:302] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:55.286407 14908 raft_consensus.cc:504] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.417520 13691 raft_consensus.cc:374] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:55.419273 14908 leader_election.cc:288] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:55.377204 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b81e7056a441d2854ba5ab6b4c5433" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:55.379237 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5794637d5dd43f2b05edd53865f4e99" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:55.421203 13062 raft_consensus.cc:2379] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in current term 1: Already voted for candidate f26517150f1c480a877f15aec71d020b in this term.
I20260810 23:32:55.417899 13691 raft_consensus.cc:729] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:55.379115 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fef127263f1470ea1056a454971e465" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:55.422156 13691 consensus_queue.cc:260] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.422230 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 330 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:55.392836 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 199) took 5810 ms. Trace:
I20260810 23:32:55.423259 14121 rpcz_store.cc:276] 0810 23:32:49.582233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.582411 (+ 178us) service_pool.cc:227] Handling call
0810 23:32:55.392338 (+5809927us) ts_tablet_manager.cc:1145] Deleting tablet 240adfa4be844518bbc9624d0e408789
0810 23:32:55.392394 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.392759 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5806827}
I20260810 23:32:55.426180 12228 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 308) took 2033 ms. Trace:
I20260810 23:32:55.427340 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 201) took 5706 ms. Trace:
I20260810 23:32:55.427785 14121 rpcz_store.cc:276] 0810 23:32:49.721023 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.721842 (+ 819us) service_pool.cc:227] Handling call
0810 23:32:55.426929 (+5705087us) ts_tablet_manager.cc:1145] Deleting tablet 840400119d20424ea0f03f2bff550d44
0810 23:32:55.426999 (+ 70us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.427261 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5694148}
I20260810 23:32:55.429257 14121 tablet_replica.cc:330] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:55.430409 14121 raft_consensus.cc:2229] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 2 LEADER]: Raft consensus shutting down.
I20260810 23:32:55.434055 14978 raft_consensus.cc:2726] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:55.435298 13665 tablet_bootstrap.cc:654] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:55.436486 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5794637d5dd43f2b05edd53865f4e99" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:55.437201 13327 raft_consensus.cc:3037] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:55.437165 12700 leader_election.cc:341] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:55.437664 12700 leader_election.cc:302] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:32:55.439388 12700 leader_election.cc:341] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:55.440840 14928 leader_election.cc:288] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:55.446573 13336 raft_consensus.cc:3037] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:55.446614 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 840400119d20424ea0f03f2bff550d44 because the tablet was not found. No further retry: Not found: Tablet not found: 840400119d20424ea0f03f2bff550d44
I20260810 23:32:55.448745 12785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5794637d5dd43f2b05edd53865f4e99" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:55.426664 12228 rpcz_store.cc:276] 0810 23:32:53.393017 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:53.715779 (+322762us) service_pool.cc:227] Handling call
0810 23:32:53.732228 (+ 16449us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7156 00007f0ce6e24034 00007f0ce6e23fc2 00007f0ce6e23ef1 00007f0ce6e23e6c 00007f0ce6e23122 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:53.735717 (+ 3489us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:53.739227 (+ 3510us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:55.424430 (+1685203us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:55.425234 (+ 804us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:55.425846 (+ 612us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:54.583701 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:54.584176 (+ 475us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:54.584203 (+ 27us) write_op.cc:435] Acquired schema lock
0810 23:32:54.584216 (+ 13us) tablet.cc:660] Decoding operations
0810 23:32:54.584544 (+ 328us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:54.584585 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:54.584611 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:54.584835 (+ 224us) tablet.cc:699] Row locks acquired
0810 23:32:54.584849 (+ 14us) write_op.cc:260] PREPARE: finished
0810 23:32:54.584979 (+ 130us) write_op.cc:270] Start()
0810 23:32:54.585074 (+ 95us) write_op.cc:276] Timestamp: P: 1786404774584923 usec, L: 0
0810 23:32:54.585092 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 23:32:54.585696 (+ 604us) log.cc:842] Serialized 433 byte log entry
0810 23:32:54.803086 (+217390us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b58000839b0. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce7291834 00007f0ce7296771 00007f0ce72966f9 00007f0ce7296689 00007f0ce7296651 00007f0ce729581d 0000000000580784 00000000005805b9 00007f0ce03c26b6 00007f0ce03c5b11
0810 23:32:55.322826 (+519740us) op_driver.cc:444] REPLICATION: finished
0810 23:32:55.411796 (+ 88970us) write_op.cc:301] APPLY: starting
0810 23:32:55.411895 (+ 99us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:55.412063 (+ 168us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:55.412083 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:55.412747 (+ 664us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:55.412910 (+ 163us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:55.412970 (+ 60us) write_op.cc:312] APPLY: finished
0810 23:32:55.413188 (+ 218us) log.cc:842] Serialized 29 byte log entry
0810 23:32:55.420351 (+ 7163us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:55.420454 (+ 103us) write_op.cc:454] Released schema lock
0810 23:32:55.420516 (+ 62us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":33582336,"child_traces":[["op",{"apply.queue_time_us":88753,"apply.run_cpu_time_us":2466,"apply.run_wall_time_us":20613,"num_ops":2,"prepare.queue_time_us":842681,"prepare.run_cpu_time_us":9424,"prepare.run_wall_time_us":175678,"raft.queue_time_us":145264,"raft.run_cpu_time_us":904,"raft.run_wall_time_us":48085,"replication_time_us":737521,"spinlock_wait_cycles":4618624,"thread_start_us":149355,"threads_started":2}]]}
I20260810 23:32:55.449875 12785 raft_consensus.cc:3037] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:55.459597 14974 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:55.460997 13691 ts_tablet_manager.cc:1428] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.169s user 0.005s sys 0.005s
I20260810 23:32:55.466243 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
W20260810 23:32:55.465523 12958 leader_election.cc:341] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: db363eb0f98d460eb6f5a577233d8ac0
W20260810 23:32:55.470043 12240 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 58) took 3363 ms (client timeout 1000 ms). Trace:
W20260810 23:32:55.470170 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 68627.3us
I20260810 23:32:55.472096 13212 leader_election.cc:302] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:55.475222 13327 raft_consensus.cc:2454] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:55.471910 13691 tablet_bootstrap.cc:492] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:55.474876 14121 pending_rounds.cc:62] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b: Trying to abort 1 pending ops.
I20260810 23:32:55.484091 14121 pending_rounds.cc:68] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7317113959485456384 op_type: NO_OP noop_request { }
I20260810 23:32:55.486127 12952 leader_election.cc:302] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:55.487399 14908 raft_consensus.cc:2781] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:55.487985 14908 raft_consensus.cc:686] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:55.484556 14121 raft_consensus.cc:2866] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:32:55.488706 14121 raft_consensus.cc:2258] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:55.489037 14908 consensus_queue.cc:237] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.439827 12700 leader_election.cc:302] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:55.497920 13665 tablet_bootstrap.cc:492] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:55.498587 13665 ts_tablet_manager.cc:1397] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.139s user 0.007s sys 0.008s
I20260810 23:32:55.501720 14121 ts_tablet_manager.cc:1905] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:55.503399 14931 raft_consensus.cc:482] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:55.503919 14931 raft_consensus.cc:504] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.506448 14931 leader_election.cc:288] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:55.509061 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01ede4b52c3498a924340b39c3c2435" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:55.509886 13335 raft_consensus.cc:2454] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:55.511577 12952 leader_election.cc:302] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:55.513029 14955 raft_consensus.cc:2726] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:55.513201 14931 raft_consensus.cc:2781] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:55.513558 14931 raft_consensus.cc:482] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:55.513901 14931 raft_consensus.cc:3037] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:55.519677 13691 tablet_bootstrap.cc:654] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:55.520731 12223 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 309) took 2020 ms. Trace:
I20260810 23:32:55.521152 12223 rpcz_store.cc:276] 0810 23:32:53.500450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:53.768690 (+268240us) service_pool.cc:227] Handling call
0810 23:32:53.773014 (+ 4324us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:53.780153 (+ 7139us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:55.519600 (+1739447us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:55.520126 (+ 526us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:55.520501 (+ 375us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:54.759579 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:54.774357 (+ 14778us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:54.774386 (+ 29us) write_op.cc:435] Acquired schema lock
0810 23:32:54.774401 (+ 15us) tablet.cc:660] Decoding operations
0810 23:32:54.774738 (+ 337us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:54.774783 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:54.774813 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:54.775116 (+ 303us) tablet.cc:699] Row locks acquired
0810 23:32:54.775137 (+ 21us) write_op.cc:260] PREPARE: finished
0810 23:32:54.775338 (+ 201us) write_op.cc:270] Start()
0810 23:32:54.775467 (+ 129us) write_op.cc:276] Timestamp: P: 1786404774775238 usec, L: 0
0810 23:32:54.775492 (+ 25us) op_driver.cc:331] REPLICATION: starting
0810 23:32:54.833046 (+ 57554us) log.cc:842] Serialized 433 byte log entry
0810 23:32:55.516410 (+683364us) op_driver.cc:444] REPLICATION: finished
0810 23:32:55.516786 (+ 376us) write_op.cc:301] APPLY: starting
0810 23:32:55.516863 (+ 77us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:55.516995 (+ 132us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:55.517128 (+ 133us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:55.517847 (+ 719us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:55.518052 (+ 205us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:55.518104 (+ 52us) write_op.cc:312] APPLY: finished
0810 23:32:55.518346 (+ 242us) log.cc:842] Serialized 29 byte log entry
0810 23:32:55.519123 (+ 777us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:55.519214 (+ 91us) write_op.cc:454] Released schema lock
0810 23:32:55.519252 (+ 38us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":129,"num_ops":2,"prepare.queue_time_us":977347,"prepare.run_cpu_time_us":3138,"prepare.run_wall_time_us":156989,"replication_time_us":740636}]]}
W20260810 23:32:55.480001 13212 leader_election.cc:341] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Not found: Tablet not found: 440de7d9de8d4be9a5322e1fe3388b9b
I20260810 23:32:55.524864 13212 leader_election.cc:302] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:55.525774 14928 raft_consensus.cc:2726] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:55.527036 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 47064us
W20260810 23:32:55.527912 13212 leader_election.cc:341] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:55.527237 13514 consensus_queue.cc:260] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.480562 13974 log.cc:1196] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/4d0d9227d8574de8bd73041f3e8043a0
I20260810 23:32:55.529769 13974 ts_tablet_manager.cc:1939] T 4d0d9227d8574de8bd73041f3e8043a0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:55.530510 13514 ts_tablet_manager.cc:1428] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.259s user 0.010s sys 0.001s
I20260810 23:32:55.535348 14877 raft_consensus.cc:2726] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:55.537688 12237 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 312) took 1612 ms. Trace:
I20260810 23:32:55.538086 12237 rpcz_store.cc:276] 0810 23:32:53.925250 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:54.386187 (+460937us) service_pool.cc:227] Handling call
0810 23:32:54.656767 (+270580us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:54.793835 (+137068us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:55.536509 (+742674us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:55.537067 (+ 558us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:55.537471 (+ 404us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:54.916861 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:54.917386 (+ 525us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:54.917421 (+ 35us) write_op.cc:435] Acquired schema lock
0810 23:32:54.917553 (+ 132us) tablet.cc:660] Decoding operations
0810 23:32:54.918179 (+ 626us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:54.918239 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:54.918271 (+ 32us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:54.918553 (+ 282us) tablet.cc:699] Row locks acquired
0810 23:32:54.918571 (+ 18us) write_op.cc:260] PREPARE: finished
0810 23:32:54.918745 (+ 174us) write_op.cc:270] Start()
0810 23:32:54.918865 (+ 120us) write_op.cc:276] Timestamp: P: 1786404774918687 usec, L: 0
0810 23:32:54.918884 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 23:32:54.919917 (+ 1033us) log.cc:842] Serialized 433 byte log entry
0810 23:32:55.533207 (+613290us) op_driver.cc:444] REPLICATION: finished
0810 23:32:55.533812 (+ 605us) write_op.cc:301] APPLY: starting
0810 23:32:55.533896 (+ 84us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:55.534051 (+ 155us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:55.534196 (+ 145us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:55.534754 (+ 558us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:55.534940 (+ 186us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:55.534989 (+ 49us) write_op.cc:312] APPLY: finished
0810 23:32:55.535232 (+ 243us) log.cc:842] Serialized 29 byte log entry
0810 23:32:55.536044 (+ 812us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:55.536142 (+ 98us) write_op.cc:454] Released schema lock
0810 23:32:55.536188 (+ 46us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":381,"num_ops":2,"prepare.queue_time_us":116957,"prepare.run_cpu_time_us":3147,"prepare.run_wall_time_us":148512,"replication_time_us":614199}]]}
I20260810 23:32:55.549403 13514 tablet_bootstrap.cc:492] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:55.549527 12224 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 310) took 1858 ms. Trace:
I20260810 23:32:55.549875 12224 rpcz_store.cc:276] 0810 23:32:53.691069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:54.358393 (+667324us) service_pool.cc:227] Handling call
0810 23:32:54.656936 (+298543us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:54.800814 (+143878us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:55.548354 (+747540us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:55.548844 (+ 490us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:55.549286 (+ 442us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:55.065600 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:55.066285 (+ 685us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:55.066324 (+ 39us) write_op.cc:435] Acquired schema lock
0810 23:32:55.066353 (+ 29us) tablet.cc:660] Decoding operations
0810 23:32:55.066836 (+ 483us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:55.066911 (+ 75us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:55.067002 (+ 91us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:55.067321 (+ 319us) tablet.cc:699] Row locks acquired
0810 23:32:55.067337 (+ 16us) write_op.cc:260] PREPARE: finished
0810 23:32:55.067539 (+ 202us) write_op.cc:270] Start()
0810 23:32:55.067676 (+ 137us) write_op.cc:276] Timestamp: P: 1786404775067449 usec, L: 0
0810 23:32:55.067702 (+ 26us) op_driver.cc:331] REPLICATION: starting
0810 23:32:55.319490 (+251788us) log.cc:842] Serialized 433 byte log entry
0810 23:32:55.533540 (+214050us) op_driver.cc:444] REPLICATION: finished
0810 23:32:55.545397 (+ 11857us) write_op.cc:301] APPLY: starting
0810 23:32:55.545587 (+ 190us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:55.545743 (+ 156us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:55.545857 (+ 114us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:55.546390 (+ 533us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:55.546561 (+ 171us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:55.546601 (+ 40us) write_op.cc:312] APPLY: finished
0810 23:32:55.546812 (+ 211us) log.cc:842] Serialized 29 byte log entry
0810 23:32:55.547751 (+ 939us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:55.547862 (+ 111us) write_op.cc:454] Released schema lock
0810 23:32:55.547904 (+ 42us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":11485,"num_ops":2,"prepare.queue_time_us":263154,"prepare.run_cpu_time_us":5266,"prepare.run_wall_time_us":254499,"raft.queue_time_us":16915,"raft.run_cpu_time_us":17,"raft.run_wall_time_us":19,"replication_time_us":465733}]]}
I20260810 23:32:55.553766 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01ede4b52c3498a924340b39c3c2435" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:55.554613 12805 raft_consensus.cc:2454] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
W20260810 23:32:55.560837 12214 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 92b81e7056a441d2854ba5ab6b4c5433 (table default.Table_e0faeb2fda9f4af48212613473c88681 [id=d8d3c66123cd4516a59a0ab8ae4d47b9], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92b81e7056a441d2854ba5ab6b4c5433 consensus state has no leader
I20260810 23:32:55.559651 12224 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_69fa732a88234f859e1cbb8d706d7c76" } new_table_name: "default.Table_ebe848b2f9ba4077a556369c87aa436a" modify_external_catalogs: true
W20260810 23:32:55.562565 12958 consensus_peers.cc:487] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd9d90e1ceab443dbec12b2c81b975e8. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:32:55.564221 12958 leader_election.cc:341] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: c3d4c879ca6a402dbafe91940e7b367a
I20260810 23:32:55.566301 12237 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_3afb998b53b2444d831fcfc512a9c665" } new_table_name: "default.Table_adde4ae3b1394291be75cc80db4f457d" modify_external_catalogs: true
W20260810 23:32:55.570374 12454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
W20260810 23:32:55.572463 12958 leader_election.cc:341] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: db363eb0f98d460eb6f5a577233d8ac0
W20260810 23:32:55.470651 12240 rpcz_store.cc:269] 0810 23:32:52.106069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.106666 (+ 597us) service_pool.cc:227] Handling call
0810 23:32:55.469743 (+3363077us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:54.576603 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:54.578921 (+ 2318us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:54.578950 (+ 29us) write_op.cc:435] Acquired schema lock
0810 23:32:54.578963 (+ 13us) tablet.cc:660] Decoding operations
0810 23:32:54.579325 (+ 362us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:54.579372 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:54.579458 (+ 86us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:54.579668 (+ 210us) tablet.cc:699] Row locks acquired
0810 23:32:54.579687 (+ 19us) write_op.cc:260] PREPARE: finished
0810 23:32:54.579843 (+ 156us) write_op.cc:270] Start()
0810 23:32:54.579957 (+ 114us) write_op.cc:276] Timestamp: P: 1786404774579779 usec, L: 0
0810 23:32:54.579976 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 23:32:54.580858 (+ 882us) log.cc:842] Serialized 536 byte log entry
0810 23:32:54.796664 (+215806us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b58000839b0. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce7291834 00007f0ce7296771 00007f0ce72966f9 00007f0ce7296689 00007f0ce7296651 00007f0ce729581d 0000000000580784 00000000005805b9 00007f0ce03c26b6 00007f0ce03c5b11
0810 23:32:55.321867 (+525203us) op_driver.cc:444] REPLICATION: finished
0810 23:32:55.402626 (+ 80759us) write_op.cc:301] APPLY: starting
0810 23:32:55.402750 (+ 124us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:55.402947 (+ 197us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:55.402983 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:55.403328 (+ 345us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:55.403551 (+ 223us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:55.403609 (+ 58us) write_op.cc:312] APPLY: finished
0810 23:32:55.403946 (+ 337us) log.cc:842] Serialized 23 byte log entry
0810 23:32:55.410927 (+ 6981us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:55.411033 (+ 106us) write_op.cc:454] Released schema lock
0810 23:32:55.411090 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:55.411358 (+ 268us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:55.411380 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":60733,"apply.run_cpu_time_us":2335,"apply.run_wall_time_us":9022,"num_ops":1,"prepare.queue_time_us":891966,"prepare.run_cpu_time_us":2865,"prepare.run_wall_time_us":6987,"raft.queue_time_us":104,"raft.run_cpu_time_us":4081,"raft.run_wall_time_us":213505,"replication_time_us":741652,"spinlock_wait_cycles":396697472}]]}
I20260810 23:32:55.569911 13691 tablet_bootstrap.cc:492] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:55.582618 13691 ts_tablet_manager.cc:1397] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.111s user 0.010s sys 0.003s
I20260810 23:32:55.594180 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 214) took 5353 ms. Trace:
I20260810 23:32:55.594659 13974 rpcz_store.cc:276] 0810 23:32:50.240850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.241008 (+ 158us) service_pool.cc:227] Handling call
0810 23:32:55.132360 (+4891352us) ts_tablet_manager.cc:1145] Deleting tablet 4d0d9227d8574de8bd73041f3e8043a0
0810 23:32:55.139721 (+ 7361us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.272375 (+132654us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:55.358792 (+ 86417us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:55.594040 (+235248us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6071,"tablet-delete.queue_time_us":4886170}
I20260810 23:32:55.591991 13691 raft_consensus.cc:348] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.596567 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 215) took 5265 ms. Trace:
I20260810 23:32:55.596720 13691 raft_consensus.cc:374] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:55.596930 13974 rpcz_store.cc:276] 0810 23:32:50.331609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.332917 (+ 1308us) service_pool.cc:227] Handling call
0810 23:32:55.596106 (+5263189us) ts_tablet_manager.cc:1145] Deleting tablet f1e8311c941b43058384cee6d8263dc1
0810 23:32:55.596177 (+ 71us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.596504 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5211086}
I20260810 23:32:55.597772 13691 raft_consensus.cc:729] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:55.598274 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 218) took 4806 ms. Trace:
I20260810 23:32:55.598606 13974 rpcz_store.cc:276] 0810 23:32:50.791860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.792048 (+ 188us) service_pool.cc:227] Handling call
0810 23:32:55.597864 (+4805816us) ts_tablet_manager.cc:1145] Deleting tablet 0be35dabbec44caebd1df687b167f34a
0810 23:32:55.597921 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.598215 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4805063}
I20260810 23:32:55.600672 13691 consensus_queue.cc:260] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:55.602130 12454 leader_election.cc:334] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:55.602594 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 681674cb24274f10b200f264f155c2d1; no voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:55.603374 14974 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260810 23:32:55.610323 13691 ts_tablet_manager.cc:1428] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
W20260810 23:32:55.616909 13212 leader_election.cc:341] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Not found: Tablet not found: 4d697e7533af4fc0a15cb41f3fcce4db
I20260810 23:32:55.617398 13212 leader_election.cc:302] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:55.619266 14877 raft_consensus.cc:2726] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 23:32:55.621007 13212 leader_election.cc:341] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:55.620771 12240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e0faeb2fda9f4af48212613473c88681" } modify_external_catalogs: true
I20260810 23:32:55.623404 13212 leader_election.cc:302] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:55.625432 14955 raft_consensus.cc:2726] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:55.632300 14877 raft_consensus.cc:2726] T 3fef127263f1470ea1056a454971e465 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:55.599736 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 223) took 3990 ms. Trace:
I20260810 23:32:55.635308 13974 rpcz_store.cc:276] 0810 23:32:51.609219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:51.628285 (+ 19066us) service_pool.cc:227] Handling call
0810 23:32:55.599476 (+3971191us) ts_tablet_manager.cc:1145] Deleting tablet fd4f55ef709d473f89a16d32160eb61b
0810 23:32:55.599522 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.599684 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3969934}
I20260810 23:32:55.636171 13691 tablet_bootstrap.cc:492] T 0d1b15dd545c4598aa5c7a9dc53618ec P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:55.615499 14931 raft_consensus.cc:504] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.640224 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 4d0d9227d8574de8bd73041f3e8043a0 (table default.Table_bf4e391ee20b45d393a269203b401bc0 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]) successfully deleted
W20260810 23:32:55.641180 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet f1e8311c941b43058384cee6d8263dc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e8311c941b43058384cee6d8263dc1 already in progress: creating tablet
W20260810 23:32:55.642045 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 0be35dabbec44caebd1df687b167f34a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0be35dabbec44caebd1df687b167f34a already in progress: creating tablet
W20260810 23:32:55.642838 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet fd4f55ef709d473f89a16d32160eb61b because the tablet was not found. No further retry: Not found: Tablet not found: fd4f55ef709d473f89a16d32160eb61b
I20260810 23:32:55.646450 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01ede4b52c3498a924340b39c3c2435" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:55.647125 13336 raft_consensus.cc:3037] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:55.652652 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 213307us
W20260810 23:32:55.653909 12700 leader_election.cc:341] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:55.654321 12700 leader_election.cc:302] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:55.655755 14955 raft_consensus.cc:2726] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:55.656975 12700 leader_election.cc:341] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:55.657346 13336 raft_consensus.cc:2454] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:55.657428 12700 leader_election.cc:302] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:55.658587 12952 leader_election.cc:302] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:55.660043 14908 raft_consensus.cc:2781] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:55.660559 14908 raft_consensus.cc:686] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:55.661465 14908 consensus_queue.cc:237] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:55.663386 12237 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7276caee8a674131860ef17898c65308 consensus state has no leader
I20260810 23:32:55.664081 12237 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 313) took 1676 ms. Trace:
I20260810 23:32:55.664261 14955 raft_consensus.cc:2726] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:55.665992 12789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01ede4b52c3498a924340b39c3c2435" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:55.666710 12789 raft_consensus.cc:3037] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:55.666520 12794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5794637d5dd43f2b05edd53865f4e99" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:55.668095 12794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01ede4b52c3498a924340b39c3c2435" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:55.668215 12240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e0faeb2fda9f4af48212613473c88681" } modify_external_catalogs: true
I20260810 23:32:55.670403 14931 leader_election.cc:288] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:55.635207 14121 ts_tablet_manager.cc:1918] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260810 23:32:55.671291 14121 log.cc:1196] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/db363eb0f98d460eb6f5a577233d8ac0
I20260810 23:32:55.683558 13062 raft_consensus.cc:3037] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:55.689612 14955 raft_consensus.cc:482] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:55.690373 14955 raft_consensus.cc:504] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.692902 14955 leader_election.cc:288] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:55.676968 12785 raft_consensus.cc:2454] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:55.692368 14121 ts_tablet_manager.cc:1939] T db363eb0f98d460eb6f5a577233d8ac0 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:55.697356 12769 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0dd19112504f8ca57f64dcf0bd50a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:55.696801 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 202) took 5961 ms. Trace:
I20260810 23:32:55.685210 13665 raft_consensus.cc:348] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.710129 13665 raft_consensus.cc:374] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:55.710408 13665 raft_consensus.cc:729] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:55.711180 13665 consensus_queue.cc:260] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.696614 14877 consensus_queue.cc:1035] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260810 23:32:55.694048 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0136a1e890436fae572b46e64750e7" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:55.716302 12698 leader_election.cc:341] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:55.688671 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 224) took 3808 ms. Trace:
I20260810 23:32:55.717906 13974 rpcz_store.cc:276] 0810 23:32:51.880463 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:51.880643 (+ 180us) service_pool.cc:227] Handling call
0810 23:32:55.638266 (+3757623us) ts_tablet_manager.cc:1145] Deleting tablet 1b1f24e226b94ccf8ad5a656c80a5b3b
0810 23:32:55.688206 (+ 49940us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.688592 (+ 386us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3756825}
W20260810 23:32:55.719311 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b1f24e226b94ccf8ad5a656c80a5b3b already in progress: creating tablet
I20260810 23:32:55.719514 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 225) took 3825 ms. Trace:
I20260810 23:32:55.719888 13974 rpcz_store.cc:276] 0810 23:32:51.894610 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:51.894772 (+ 162us) service_pool.cc:227] Handling call
0810 23:32:55.719027 (+3824255us) ts_tablet_manager.cc:1145] Deleting tablet 4d1d3bd06a9a4627a1a7e83b22dba2af
0810 23:32:55.719079 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.719430 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3823523}
W20260810 23:32:55.721648 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 4d1d3bd06a9a4627a1a7e83b22dba2af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d1d3bd06a9a4627a1a7e83b22dba2af already in progress: creating tablet
I20260810 23:32:55.721872 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 226) took 3760 ms. Trace:
I20260810 23:32:55.722397 13974 rpcz_store.cc:276] 0810 23:32:51.961537 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:51.961750 (+ 213us) service_pool.cc:227] Handling call
0810 23:32:55.721430 (+3759680us) ts_tablet_manager.cc:1145] Deleting tablet a480093c481941f0bb35949c7606f37f
0810 23:32:55.721486 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.721806 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3758757}
I20260810 23:32:55.723667 13974 tablet_replica.cc:330] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
W20260810 23:32:55.723694 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet a480093c481941f0bb35949c7606f37f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a480093c481941f0bb35949c7606f37f already in progress: creating tablet
I20260810 23:32:55.724900 13974 raft_consensus.cc:2229] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 23:32:55.632750 14989 log.cc:924] Time spent T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.126s user 0.001s sys 0.000s
I20260810 23:32:55.744423 15008 tablet_replica.cc:330] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:55.634567 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01ede4b52c3498a924340b39c3c2435" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:55.700642 14121 rpcz_store.cc:276] 0810 23:32:49.735604 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.736130 (+ 526us) service_pool.cc:227] Handling call
0810 23:32:55.429025 (+5692895us) ts_tablet_manager.cc:1145] Deleting tablet db363eb0f98d460eb6f5a577233d8ac0
0810 23:32:55.429085 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.561234 (+132149us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:55.634921 (+ 73687us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:55.696714 (+ 61793us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43427,"tablet-delete.queue_time_us":5660411}
I20260810 23:32:55.753777 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet db363eb0f98d460eb6f5a577233d8ac0 (table default.Table_e81793ce40c14cfe81a319f4cbd088bf [id=cca554c88a7844d5aa3312b12e89369e]) successfully deleted
I20260810 23:32:55.755734 12789 raft_consensus.cc:2454] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:55.757025 14121 tablet_replica.cc:330] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:55.759001 15008 raft_consensus.cc:2229] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:55.664386 12237 rpcz_store.cc:276] 0810 23:32:53.987264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:55.544011 (+1556747us) service_pool.cc:227] Handling call
0810 23:32:55.566185 (+ 22174us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e77d6 00007f0ce6e2d634 00007f0ce6e2d5c2 00007f0ce6e2d4f1 00007f0ce6e2d46c 00007f0ce6e2c722 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:55.567217 (+ 1032us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:55.568163 (+ 946us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:55.568965 (+ 802us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:55.569228 (+ 263us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:55.620379 (+ 51151us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:55.663979 (+ 43600us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:55.570826 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:55.571429 (+ 603us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:55.571488 (+ 59us) write_op.cc:435] Acquired schema lock
0810 23:32:55.571509 (+ 21us) tablet.cc:660] Decoding operations
0810 23:32:55.571913 (+ 404us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:55.572056 (+ 143us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:55.572138 (+ 82us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:55.572516 (+ 378us) tablet.cc:699] Row locks acquired
0810 23:32:55.572562 (+ 46us) write_op.cc:260] PREPARE: finished
0810 23:32:55.572780 (+ 218us) write_op.cc:270] Start()
0810 23:32:55.572919 (+ 139us) write_op.cc:276] Timestamp: P: 1786404775572648 usec, L: 0
0810 23:32:55.572936 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:32:55.573907 (+ 971us) log.cc:842] Serialized 405 byte log entry
0810 23:32:55.608991 (+ 35084us) op_driver.cc:444] REPLICATION: finished
0810 23:32:55.610684 (+ 1693us) write_op.cc:301] APPLY: starting
0810 23:32:55.610769 (+ 85us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:55.610881 (+ 112us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:55.611054 (+ 173us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:55.611260 (+ 206us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:55.611396 (+ 136us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:55.611441 (+ 45us) write_op.cc:312] APPLY: finished
0810 23:32:55.611658 (+ 217us) log.cc:842] Serialized 23 byte log entry
0810 23:32:55.612445 (+ 787us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:55.612521 (+ 76us) write_op.cc:454] Released schema lock
0810 23:32:55.612556 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:55.612755 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:55.612781 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":48163072,"child_traces":[["op",{"apply.queue_time_us":1455,"apply.run_cpu_time_us":1683,"apply.run_wall_time_us":2288,"num_ops":1,"prepare.queue_time_us":178,"prepare.run_cpu_time_us":3541,"prepare.run_wall_time_us":3540,"raft.queue_time_us":125,"raft.run_cpu_time_us":909,"raft.run_wall_time_us":911,"replication_time_us":35921,"spinlock_wait_cycles":21376}]]}
I20260810 23:32:55.772241 12237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_3b3f1f30941b41d1a42bad9834394fe9" } modify_external_catalogs: true
W20260810 23:32:55.778025 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 62378.9us
I20260810 23:32:55.778770 13665 ts_tablet_manager.cc:1428] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.280s user 0.005s sys 0.002s
I20260810 23:32:55.779228 13213 leader_election.cc:302] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:55.782421 14977 raft_consensus.cc:2726] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:55.785770 13974 raft_consensus.cc:2258] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:55.792543 12237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_3b3f1f30941b41d1a42bad9834394fe9" } modify_external_catalogs: true
I20260810 23:32:55.758294 14121 raft_consensus.cc:2229] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:55.801026 14121 raft_consensus.cc:2258] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:55.802425 13974 ts_tablet_manager.cc:1905] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:55.805274 13212 leader_election.cc:302] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:32:55.806658 14877 raft_consensus.cc:2726] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:55.806668 15008 raft_consensus.cc:2258] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:55.779914 13665 tablet_bootstrap.cc:492] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:55.815622 13691 tablet_bootstrap.cc:654] T 0d1b15dd545c4598aa5c7a9dc53618ec P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:55.815575 14121 ts_tablet_manager.cc:1905] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:55.819864 14999 raft_consensus.cc:482] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:55.820380 14999 raft_consensus.cc:504] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.822520 14999 leader_election.cc:288] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:55.831535 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0136a1e890436fae572b46e64750e7" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:55.840521 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdec13ee70d402fa3a08119e7bf1313" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:55.841421 13327 raft_consensus.cc:2454] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
W20260810 23:32:55.845877 12700 leader_election.cc:341] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:55.846372 12700 leader_election.cc:302] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:55.847589 12952 leader_election.cc:302] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:55.851713 15014 raft_consensus.cc:2781] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:55.852353 15014 raft_consensus.cc:482] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:55.855325 15014 raft_consensus.cc:3037] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:55.867028 14877 consensus_queue.cc:1035] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:55.885406 13691 tablet_bootstrap.cc:492] T 0d1b15dd545c4598aa5c7a9dc53618ec P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:55.885903 13691 ts_tablet_manager.cc:1397] T 0d1b15dd545c4598aa5c7a9dc53618ec P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.250s user 0.010s sys 0.008s
I20260810 23:32:55.888441 13691 raft_consensus.cc:348] T 0d1b15dd545c4598aa5c7a9dc53618ec P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.889139 13691 raft_consensus.cc:374] T 0d1b15dd545c4598aa5c7a9dc53618ec P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:55.889379 13691 raft_consensus.cc:729] T 0d1b15dd545c4598aa5c7a9dc53618ec P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:55.890023 13691 consensus_queue.cc:260] T 0d1b15dd545c4598aa5c7a9dc53618ec P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.891737 13691 ts_tablet_manager.cc:1428] T 0d1b15dd545c4598aa5c7a9dc53618ec P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:32:55.892680 13691 tablet_bootstrap.cc:492] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:55.894532 15014 raft_consensus.cc:504] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.901473 13665 tablet_bootstrap.cc:654] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:55.901598 15008 ts_tablet_manager.cc:1905] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:55.904490 13514 tablet_bootstrap.cc:654] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:55.904603 14985 raft_consensus.cc:2726] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:55.863802 12789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdec13ee70d402fa3a08119e7bf1313" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:55.905951 12789 raft_consensus.cc:2454] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:55.897768 14978 raft_consensus.cc:482] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:55.907034 14978 raft_consensus.cc:504] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.926528 15014 leader_election.cc:288] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:55.927820 14978 leader_election.cc:288] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:55.937847 13514 tablet_bootstrap.cc:492] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:55.938189 13974 ts_tablet_manager.cc:1918] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:55.938642 13974 log.cc:1196] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/fd9d90e1ceab443dbec12b2c81b975e8
I20260810 23:32:55.938743 13514 ts_tablet_manager.cc:1397] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.390s user 0.012s sys 0.011s
I20260810 23:32:55.939623 13974 ts_tablet_manager.cc:1939] T fd9d90e1ceab443dbec12b2c81b975e8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:55.943864 13514 raft_consensus.cc:348] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.951815 13514 raft_consensus.cc:374] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:55.952868 13514 raft_consensus.cc:729] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:55.957872 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47b8b57d5b44a018bc2d7b3f976d803" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:55.959091 13514 consensus_queue.cc:260] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.959450 12785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdec13ee70d402fa3a08119e7bf1313" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:55.964310 12785 raft_consensus.cc:3037] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:55.964999 13514 ts_tablet_manager.cc:1428] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.026s user 0.006s sys 0.002s
I20260810 23:32:55.962004 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 227) took 3913 ms. Trace:
W20260810 23:32:55.962795 12700 leader_election.cc:341] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:55.975118 13974 rpcz_store.cc:276] 0810 23:32:52.048215 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.048572 (+ 357us) service_pool.cc:227] Handling call
0810 23:32:55.723510 (+3674938us) ts_tablet_manager.cc:1145] Deleting tablet fd9d90e1ceab443dbec12b2c81b975e8
0810 23:32:55.723551 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.901022 (+177471us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:55.937882 (+ 36860us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:55.961910 (+ 24028us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60085,"tablet-delete.queue_time_us":3650602}
I20260810 23:32:55.976459 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet fd9d90e1ceab443dbec12b2c81b975e8 (table default.Table_dc5d83ea7f1441fa9e2b2536bab49724 [id=aef6fd3aebac448e96d5942fd12836ca]) successfully deleted
I20260810 23:32:55.977252 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 228) took 3602 ms. Trace:
I20260810 23:32:55.960420 14978 raft_consensus.cc:482] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:55.978528 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdec13ee70d402fa3a08119e7bf1313" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:55.978844 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47b8b57d5b44a018bc2d7b3f976d803" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:55.979210 13334 raft_consensus.cc:3037] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:55.978786 14978 raft_consensus.cc:504] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.981984 14978 leader_election.cc:288] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:55.983453 13514 tablet_bootstrap.cc:492] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:55.983392 14121 ts_tablet_manager.cc:1918] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:55.985002 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3049bcdc6e43c795961c74f0d008f5" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:55.986821 14977 raft_consensus.cc:482] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:32:55.988395 12698 leader_election.cc:341] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:55.988967 12698 leader_election.cc:302] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:55.987411 14977 raft_consensus.cc:504] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:55.993883 14977 leader_election.cc:288] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:55.997090 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d1b15dd545c4598aa5c7a9dc53618ec" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:55.997946 13054 raft_consensus.cc:2454] T 0d1b15dd545c4598aa5c7a9dc53618ec P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:55.994072 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d1b15dd545c4598aa5c7a9dc53618ec" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:55.999359 12806 raft_consensus.cc:2454] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:55.999500 14978 raft_consensus.cc:2726] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:56.000689 13213 leader_election.cc:302] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:56.001718 14977 raft_consensus.cc:2781] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:56.002106 14977 raft_consensus.cc:482] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:56.002050 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3049bcdc6e43c795961c74f0d008f5" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:56.002259 13334 raft_consensus.cc:2454] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:56.002534 14977 raft_consensus.cc:3037] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:55.977551 13974 rpcz_store.cc:276] 0810 23:32:52.374629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.380676 (+ 6047us) service_pool.cc:227] Handling call
0810 23:32:55.976813 (+3596137us) ts_tablet_manager.cc:1145] Deleting tablet d603569c54ef4ffba9941d0cc3e0f97f
0810 23:32:55.976866 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.977187 (+ 321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3595368}
W20260810 23:32:56.003539 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet d603569c54ef4ffba9941d0cc3e0f97f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d603569c54ef4ffba9941d0cc3e0f97f already in progress: creating tablet
I20260810 23:32:56.004477 13974 tablet_replica.cc:330] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
W20260810 23:32:56.003959 12700 leader_election.cc:341] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:56.005371 13974 raft_consensus.cc:2229] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:56.008306 13665 tablet_bootstrap.cc:492] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:56.009268 12952 leader_election.cc:302] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:56.010308 15014 raft_consensus.cc:2781] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:56.011157 15014 raft_consensus.cc:686] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:56.012185 15014 consensus_queue.cc:237] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:56.021788 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 53164.6us
I20260810 23:32:56.025949 13665 ts_tablet_manager.cc:1397] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.246s user 0.008s sys 0.006s
I20260810 23:32:56.008354 13974 raft_consensus.cc:2258] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:55.983848 14121 log.cc:1196] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/42517f11422b42c39db282e2c5f584a3
I20260810 23:32:56.030269 14121 ts_tablet_manager.cc:1939] T 42517f11422b42c39db282e2c5f584a3 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:56.030685 13974 ts_tablet_manager.cc:1905] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:56.037978 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 203) took 6245 ms. Trace:
I20260810 23:32:56.038396 14121 rpcz_store.cc:276] 0810 23:32:49.792839 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:49.793137 (+ 298us) service_pool.cc:227] Handling call
0810 23:32:55.755401 (+5962264us) ts_tablet_manager.cc:1145] Deleting tablet 42517f11422b42c39db282e2c5f584a3
0810 23:32:55.755457 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:55.835236 (+ 79779us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:55.983070 (+147834us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:56.037887 (+ 54817us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31872,"tablet-delete.queue_time_us":5961335}
I20260810 23:32:56.040215 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 42517f11422b42c39db282e2c5f584a3 (table default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f]) successfully deleted
I20260810 23:32:56.043871 13691 tablet_bootstrap.cc:654] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:56.051870 14977 raft_consensus.cc:504] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:56.054989 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 66635.1us
W20260810 23:32:56.055817 12698 leader_election.cc:341] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:56.056223 12698 leader_election.cc:302] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:56.056491 14977 leader_election.cc:288] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:56.057621 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 204) took 5782 ms. Trace:
I20260810 23:32:56.057258 14978 raft_consensus.cc:2726] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:56.058015 14121 rpcz_store.cc:276] 0810 23:32:50.275006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.277266 (+ 2260us) service_pool.cc:227] Handling call
0810 23:32:56.039692 (+5762426us) ts_tablet_manager.cc:1145] Deleting tablet 4d0d9227d8574de8bd73041f3e8043a0
0810 23:32:56.039736 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:56.057531 (+ 17795us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5761424}
W20260810 23:32:56.060990 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 4d0d9227d8574de8bd73041f3e8043a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d0d9227d8574de8bd73041f3e8043a0 already in progress: creating tablet
I20260810 23:32:56.061790 13665 raft_consensus.cc:348] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.062492 13665 raft_consensus.cc:374] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:56.062721 13665 raft_consensus.cc:729] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:56.063301 13665 consensus_queue.cc:260] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.090489 13514 tablet_bootstrap.cc:654] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:56.092324 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d1b15dd545c4598aa5c7a9dc53618ec" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:56.093075 13054 raft_consensus.cc:3037] T 0d1b15dd545c4598aa5c7a9dc53618ec P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:56.098431 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d1b15dd545c4598aa5c7a9dc53618ec" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:56.099073 12806 raft_consensus.cc:3037] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:56.100862 15008 ts_tablet_manager.cc:1918] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:56.101301 15008 log.cc:1196] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/ca0dd19112504f8ca57f64dcf0bd50a8
I20260810 23:32:56.106086 15008 ts_tablet_manager.cc:1939] T ca0dd19112504f8ca57f64dcf0bd50a8 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:56.062186 14121 tablet_replica.cc:330] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:56.107852 14121 raft_consensus.cc:2229] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:56.112815 14121 raft_consensus.cc:2258] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:56.115074 14121 ts_tablet_manager.cc:1905] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:56.123340 13054 raft_consensus.cc:2454] T 0d1b15dd545c4598aa5c7a9dc53618ec P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:56.134747 13212 leader_election.cc:302] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:56.135778 14977 raft_consensus.cc:2781] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:56.136484 14977 raft_consensus.cc:686] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:56.137413 14977 consensus_queue.cc:237] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.143074 13665 ts_tablet_manager.cc:1428] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.117s user 0.003s sys 0.003s
I20260810 23:32:56.146577 13691 tablet_bootstrap.cc:492] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:56.147102 13691 ts_tablet_manager.cc:1397] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.255s user 0.011s sys 0.000s
I20260810 23:32:56.151638 13665 tablet_bootstrap.cc:492] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:56.182091 12806 raft_consensus.cc:2454] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
W20260810 23:32:56.187076 12958 consensus_peers.cc:487] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca0dd19112504f8ca57f64dcf0bd50a8. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:56.189322 12785 raft_consensus.cc:2454] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:56.191431 13514 tablet_bootstrap.cc:492] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:56.191915 13514 ts_tablet_manager.cc:1397] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.209s user 0.025s sys 0.016s
I20260810 23:32:56.194821 13514 raft_consensus.cc:348] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.195492 13514 raft_consensus.cc:374] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:56.195731 13514 raft_consensus.cc:729] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
W20260810 23:32:56.201419 12958 consensus_peers.cc:487] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3d4c879ca6a402dbafe91940e7b367a. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:56.204535 13514 consensus_queue.cc:260] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.206777 13514 ts_tablet_manager.cc:1428] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20260810 23:32:56.207840 13514 tablet_bootstrap.cc:492] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:56.234050 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet ca0dd19112504f8ca57f64dcf0bd50a8 (table default.Table_1e5f07908d444cf9af2cae2ea83ae7ae [id=a591a34da3514b87aa852684669a399f]) successfully deleted
I20260810 23:32:56.240444 15042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e964330cf113403a8aa5b73ddd3b31d0)
I20260810 23:32:56.241835 13514 tablet_bootstrap.cc:654] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:56.241364 15042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:56.249559 13691 raft_consensus.cc:348] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:56.250458 13691 raft_consensus.cc:374] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:56.250758 13691 raft_consensus.cc:729] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:56.251647 13691 consensus_queue.cc:260] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:56.251499 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 332 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
W20260810 23:32:56.280105 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 95928us
I20260810 23:32:56.285385 15042 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:56.293318 15011 consensus_queue.cc:1035] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:56.294866 13691 ts_tablet_manager.cc:1428] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.147s user 0.009s sys 0.000s
I20260810 23:32:56.298619 13691 tablet_bootstrap.cc:492] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:56.305886 13514 tablet_bootstrap.cc:492] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:56.306681 13514 ts_tablet_manager.cc:1397] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.099s user 0.016s sys 0.002s
I20260810 23:32:56.309842 13514 raft_consensus.cc:348] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:56.314764 15011 raft_consensus.cc:482] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:32:56.314883 12454 leader_election.cc:334] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:56.315480 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 681674cb24274f10b200f264f155c2d1; no voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:56.315366 15011 raft_consensus.cc:504] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:56.316541 15042 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260810 23:32:56.320719 15011 leader_election.cc:288] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:56.323585 13974 ts_tablet_manager.cc:1918] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:56.324036 13974 log.cc:1196] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/db363eb0f98d460eb6f5a577233d8ac0
I20260810 23:32:56.324759 13974 ts_tablet_manager.cc:1939] T db363eb0f98d460eb6f5a577233d8ac0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:56.319972 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0d9227d8574de8bd73041f3e8043a0" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:56.325452 13514 raft_consensus.cc:374] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:56.335626 13514 raft_consensus.cc:729] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:56.325577 12806 raft_consensus.cc:2454] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:56.338025 13514 consensus_queue.cc:260] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:56.338551 12958 leader_election.cc:302] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:32:56.339520 15013 raft_consensus.cc:2781] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:56.339893 15013 raft_consensus.cc:482] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:56.340220 15013 raft_consensus.cc:3037] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:56.349591 13665 tablet_bootstrap.cc:654] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:56.352156 15013 raft_consensus.cc:504] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:56.355535 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0d9227d8574de8bd73041f3e8043a0" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:56.356251 12806 raft_consensus.cc:3037] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:56.358610 14985 raft_consensus.cc:482] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:56.359115 14985 raft_consensus.cc:504] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.358790 13514 ts_tablet_manager.cc:1428] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.052s user 0.007s sys 0.001s
I20260810 23:32:56.361641 14985 leader_election.cc:288] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:56.363842 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd24471dd52149d395808493ad9fa68c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:56.365057 13691 tablet_bootstrap.cc:654] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:56.366748 12700 leader_election.cc:341] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:56.369004 14121 ts_tablet_manager.cc:1918] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:56.372392 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd24471dd52149d395808493ad9fa68c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:56.374418 12698 leader_election.cc:341] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:56.374891 12698 leader_election.cc:302] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:56.376639 13514 tablet_bootstrap.cc:492] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:56.377283 14985 raft_consensus.cc:2726] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:56.386407 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 229) took 3900 ms. Trace:
I20260810 23:32:56.391310 13514 tablet_bootstrap.cc:654] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:56.371402 12952 leader_election.cc:341] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 4d0d9227d8574de8bd73041f3e8043a0
I20260810 23:32:56.391918 14121 log.cc:1196] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/13ad5d63a2104995990a86a000963cdc
I20260810 23:32:56.392829 14121 ts_tablet_manager.cc:1939] T 13ad5d63a2104995990a86a000963cdc P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:56.395203 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 205) took 6102 ms. Trace:
I20260810 23:32:56.401631 15011 consensus_queue.cc:1035] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:32:56.410269 12218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.188:45477, user_credentials={real_user=slave}} blocked reactor thread for 85490.5us
I20260810 23:32:56.412457 15013 leader_election.cc:288] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
W20260810 23:32:56.417402 12952 leader_election.cc:341] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 4d0d9227d8574de8bd73041f3e8043a0
I20260810 23:32:56.418598 12806 raft_consensus.cc:2454] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:56.420845 13974 rpcz_store.cc:276] 0810 23:32:52.486344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.486496 (+ 152us) service_pool.cc:227] Handling call
0810 23:32:56.004285 (+3517789us) ts_tablet_manager.cc:1145] Deleting tablet db363eb0f98d460eb6f5a577233d8ac0
0810 23:32:56.004339 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:56.153142 (+148803us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:56.323303 (+170161us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:56.386300 (+ 62997us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6827,"tablet-delete.queue_time_us":3516616}
I20260810 23:32:56.422214 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet db363eb0f98d460eb6f5a577233d8ac0 (table default.Table_e81793ce40c14cfe81a319f4cbd088bf [id=cca554c88a7844d5aa3312b12e89369e]) successfully deleted
I20260810 23:32:56.430532 14985 raft_consensus.cc:482] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:56.431093 14985 raft_consensus.cc:504] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.432886 12958 leader_election.cc:302] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:56.433568 14985 leader_election.cc:288] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:56.434125 15014 raft_consensus.cc:2781] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:56.434628 15014 raft_consensus.cc:686] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:56.435235 13691 tablet_bootstrap.cc:492] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:56.435820 13691 ts_tablet_manager.cc:1397] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.137s user 0.010s sys 0.003s
I20260810 23:32:56.435546 15014 consensus_queue.cc:237] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:56.441790 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41ddf9f6f71443ba75c97d2b010372c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:56.444990 12700 leader_election.cc:341] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:56.444212 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41ddf9f6f71443ba75c97d2b010372c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:56.446543 15014 consensus_queue.cc:1035] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260810 23:32:56.450371 12698 leader_election.cc:341] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:56.450886 12698 leader_election.cc:302] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:56.452353 14985 raft_consensus.cc:2726] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:56.453217 14979 raft_consensus.cc:482] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:56.453774 14979 raft_consensus.cc:504] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:56.458339 14979 leader_election.cc:288] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:56.459587 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b9d20d09c4d1ba9a4b10959807e50" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:56.459748 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b9d20d09c4d1ba9a4b10959807e50" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:32:56.461510 12698 leader_election.cc:341] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:56.462023 12700 leader_election.cc:341] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:56.462265 13514 tablet_bootstrap.cc:492] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:56.462417 12700 leader_election.cc:302] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:56.462714 13514 ts_tablet_manager.cc:1397] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.004s
I20260810 23:32:56.463482 14979 raft_consensus.cc:2726] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:56.466480 13665 tablet_bootstrap.cc:492] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:56.467308 13665 ts_tablet_manager.cc:1397] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.323s user 0.016s sys 0.000s
I20260810 23:32:56.470499 13665 raft_consensus.cc:348] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.395571 14121 rpcz_store.cc:276] 0810 23:32:50.292693 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.292869 (+ 176us) service_pool.cc:227] Handling call
0810 23:32:56.061997 (+5769128us) ts_tablet_manager.cc:1145] Deleting tablet 13ad5d63a2104995990a86a000963cdc
0810 23:32:56.062045 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:56.295298 (+233253us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:56.368699 (+ 73401us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:56.395123 (+ 26424us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4281,"tablet-delete.queue_time_us":5767633}
I20260810 23:32:56.479805 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 13ad5d63a2104995990a86a000963cdc (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5]) successfully deleted
I20260810 23:32:56.480286 13691 raft_consensus.cc:348] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.481072 13691 raft_consensus.cc:374] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:56.481324 13691 raft_consensus.cc:729] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:56.482051 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 206) took 6182 ms. Trace:
I20260810 23:32:56.483729 13514 raft_consensus.cc:348] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.484505 13514 raft_consensus.cc:374] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:56.484750 13514 raft_consensus.cc:729] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:56.485383 13514 consensus_queue.cc:260] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.486914 13514 ts_tablet_manager.cc:1428] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260810 23:32:56.491422 13665 raft_consensus.cc:374] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:56.491959 13665 raft_consensus.cc:729] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:56.477663 13974 tablet_replica.cc:330] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:56.502975 13974 raft_consensus.cc:2229] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:56.503820 13974 raft_consensus.cc:2258] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:56.506561 13974 ts_tablet_manager.cc:1905] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:56.482585 14121 rpcz_store.cc:276] 0810 23:32:50.299404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.299617 (+ 213us) service_pool.cc:227] Handling call
0810 23:32:56.481413 (+6181796us) ts_tablet_manager.cc:1145] Deleting tablet f1e8311c941b43058384cee6d8263dc1
0810 23:32:56.481482 (+ 69us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:56.481962 (+ 480us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6180529}
I20260810 23:32:56.488006 13514 tablet_bootstrap.cc:492] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:32:56.518420 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet f1e8311c941b43058384cee6d8263dc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e8311c941b43058384cee6d8263dc1 already in progress: creating tablet
I20260810 23:32:56.496732 13691 consensus_queue.cc:260] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.523497 13691 ts_tablet_manager.cc:1428] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.087s user 0.006s sys 0.000s
I20260810 23:32:56.524904 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 7293fddc208a4ee7ad85906f28f3be08 (DEFAULT_TABLE table=default.Table_9bbb5f34eb6d4d65995978e1c125a8bd [id=fbf5e7c5b1a949a3a645b7d50cbf95cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.526322 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7293fddc208a4ee7ad85906f28f3be08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.537335 12229 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:56.538141 14121 tablet_replica.cc:330] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:56.538103 13014 tablet_service.cc:1467] Processing CreateTablet for tablet 7293fddc208a4ee7ad85906f28f3be08 (DEFAULT_TABLE table=default.Table_9bbb5f34eb6d4d65995978e1c125a8bd [id=fbf5e7c5b1a949a3a645b7d50cbf95cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.542912 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7293fddc208a4ee7ad85906f28f3be08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.546448 12764 tablet_service.cc:1467] Processing CreateTablet for tablet a2a0aaa2495a4acc8ccfa26aad5b02c3 (DEFAULT_TABLE table=default.Table_b3a3530a31c5486b9502e2fbba173887 [id=84ef6dc51476443392a1adc3bc66c853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.547729 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2a0aaa2495a4acc8ccfa26aad5b02c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.552912 14121 raft_consensus.cc:2229] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:56.554325 14121 raft_consensus.cc:2258] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:56.559903 12229 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 92b81e7056a441d2854ba5ab6b4c5433 (table default.Table_e0faeb2fda9f4af48212613473c88681 [id=d8d3c66123cd4516a59a0ab8ae4d47b9]). Expected version 2 got 0
I20260810 23:32:56.563699 13023 tablet_service.cc:1467] Processing CreateTablet for tablet a2a0aaa2495a4acc8ccfa26aad5b02c3 (DEFAULT_TABLE table=default.Table_b3a3530a31c5486b9502e2fbba173887 [id=84ef6dc51476443392a1adc3bc66c853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.565299 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2a0aaa2495a4acc8ccfa26aad5b02c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.566321 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 54c9310d98ad4d20b9a584bde8525227 (DEFAULT_TABLE table=default.Table_a8675270899846aaa4209e810eb7614b [id=9e4c719f0c7a4f839aed37bc54648a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.567549 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c9310d98ad4d20b9a584bde8525227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.580993 13691 tablet_bootstrap.cc:492] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:56.540652 13278 tablet_service.cc:1467] Processing CreateTablet for tablet 7293fddc208a4ee7ad85906f28f3be08 (DEFAULT_TABLE table=default.Table_9bbb5f34eb6d4d65995978e1c125a8bd [id=fbf5e7c5b1a949a3a645b7d50cbf95cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.581918 13514 tablet_bootstrap.cc:654] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:56.582976 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7293fddc208a4ee7ad85906f28f3be08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.532907 13665 consensus_queue.cc:260] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.571377 13279 tablet_service.cc:1467] Processing CreateTablet for tablet a2a0aaa2495a4acc8ccfa26aad5b02c3 (DEFAULT_TABLE table=default.Table_b3a3530a31c5486b9502e2fbba173887 [id=84ef6dc51476443392a1adc3bc66c853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.602715 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2a0aaa2495a4acc8ccfa26aad5b02c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.621641 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 54c9310d98ad4d20b9a584bde8525227 (DEFAULT_TABLE table=default.Table_a8675270899846aaa4209e810eb7614b [id=9e4c719f0c7a4f839aed37bc54648a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.622507 13665 ts_tablet_manager.cc:1428] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.155s user 0.007s sys 0.000s
I20260810 23:32:56.622774 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c9310d98ad4d20b9a584bde8525227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.597190 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 54c9310d98ad4d20b9a584bde8525227 (DEFAULT_TABLE table=default.Table_a8675270899846aaa4209e810eb7614b [id=9e4c719f0c7a4f839aed37bc54648a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.624895 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c9310d98ad4d20b9a584bde8525227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.664819 13665 tablet_bootstrap.cc:492] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:56.675727 12237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 5d96bc27a87a41908f3ec838c228030d
I20260810 23:32:56.675581 13008 tablet_service.cc:1467] Processing CreateTablet for tablet 474a6165ce1440a294c4878602f0aed3 (DEFAULT_TABLE table=default.Table_ac935ba0ebf44788b4f41765bc914fe9 [id=aeb6bf538aa84a7fadd38df9ddb652c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.677131 13008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474a6165ce1440a294c4878602f0aed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.678897 12237 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 316) took 2081 ms. Trace:
I20260810 23:32:56.678951 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d96bc27a87a41908f3ec838c228030d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:55 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:56.683830 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d96bc27a87a41908f3ec838c228030d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:55 UTC) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:56.688562 12952 consensus_peers.cc:487] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd9d90e1ceab443dbec12b2c81b975e8. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:56.679247 12237 rpcz_store.cc:276] 0810 23:32:54.597114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:55.771789 (+1174675us) service_pool.cc:227] Handling call
0810 23:32:55.793777 (+ 21988us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:55.795384 (+ 1607us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:55.795850 (+ 466us) catalog_manager.cc:2761] Locking tablets
0810 23:32:56.520689 (+724839us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:56.675194 (+154505us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:56.675498 (+ 304us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:56.675620 (+ 122us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:56.678814 (+ 3194us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:56.524230 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:56.524700 (+ 470us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:56.524726 (+ 26us) write_op.cc:435] Acquired schema lock
0810 23:32:56.524743 (+ 17us) tablet.cc:660] Decoding operations
0810 23:32:56.525142 (+ 399us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:56.525196 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:56.525224 (+ 28us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:56.525525 (+ 301us) tablet.cc:699] Row locks acquired
0810 23:32:56.525544 (+ 19us) write_op.cc:260] PREPARE: finished
0810 23:32:56.525687 (+ 143us) write_op.cc:270] Start()
0810 23:32:56.525788 (+ 101us) write_op.cc:276] Timestamp: P: 1786404776525638 usec, L: 0
0810 23:32:56.525801 (+ 13us) op_driver.cc:331] REPLICATION: starting
0810 23:32:56.526428 (+ 627us) log.cc:842] Serialized 734 byte log entry
0810 23:32:56.570508 (+ 44080us) op_driver.cc:444] REPLICATION: finished
0810 23:32:56.617312 (+ 46804us) write_op.cc:301] APPLY: starting
0810 23:32:56.619785 (+ 2473us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:56.619908 (+ 123us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:56.620013 (+ 105us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:56.620395 (+ 382us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:56.620595 (+ 200us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:56.620657 (+ 62us) write_op.cc:312] APPLY: finished
0810 23:32:56.620923 (+ 266us) log.cc:842] Serialized 29 byte log entry
0810 23:32:56.655161 (+ 34238us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:56.655280 (+ 119us) write_op.cc:454] Released schema lock
0810 23:32:56.655329 (+ 49us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":46597,"num_ops":2,"prepare.queue_time_us":1064,"prepare.run_cpu_time_us":2581,"prepare.run_wall_time_us":2581,"raft.queue_time_us":1209,"raft.run_cpu_time_us":680,"raft.run_wall_time_us":680,"replication_time_us":44611,"thread_start_us":82417,"threads_started":1}]]}
I20260810 23:32:56.706475 14121 ts_tablet_manager.cc:1905] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:56.730530 13691 tablet_bootstrap.cc:654] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:56.733929 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 474a6165ce1440a294c4878602f0aed3 (DEFAULT_TABLE table=default.Table_ac935ba0ebf44788b4f41765bc914fe9 [id=aeb6bf538aa84a7fadd38df9ddb652c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.745693 14977 raft_consensus.cc:482] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:56.746264 14977 raft_consensus.cc:504] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.750025 14977 leader_election.cc:288] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:56.750887 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47b8b57d5b44a018bc2d7b3f976d803" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:56.751612 13054 raft_consensus.cc:2454] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:56.753847 13212 leader_election.cc:302] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:56.755316 14977 raft_consensus.cc:2781] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:56.755735 14977 raft_consensus.cc:482] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:56.756206 14977 raft_consensus.cc:3037] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:56.765013 13665 tablet_bootstrap.cc:654] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:56.768451 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474a6165ce1440a294c4878602f0aed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.778950 13279 tablet_service.cc:1467] Processing CreateTablet for tablet 474a6165ce1440a294c4878602f0aed3 (DEFAULT_TABLE table=default.Table_ac935ba0ebf44788b4f41765bc914fe9 [id=aeb6bf538aa84a7fadd38df9ddb652c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.732834 14985 raft_consensus.cc:482] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:56.781587 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474a6165ce1440a294c4878602f0aed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.788275 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47b8b57d5b44a018bc2d7b3f976d803" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:56.789369 12806 raft_consensus.cc:2454] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:56.788856 14977 raft_consensus.cc:504] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.791724 13514 tablet_bootstrap.cc:492] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:56.792202 13514 ts_tablet_manager.cc:1397] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.304s user 0.019s sys 0.001s
I20260810 23:32:56.796998 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47b8b57d5b44a018bc2d7b3f976d803" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:56.797819 13054 raft_consensus.cc:3037] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:56.817783 14977 leader_election.cc:288] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:56.810245 13054 raft_consensus.cc:2454] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:56.823833 12223 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
W20260810 23:32:56.824265 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 7276caee8a674131860ef17898c65308 (table default.Table_adde4ae3b1394291be75cc80db4f457d [id=a02006db831d417597d63c0e7c9cad8c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7276caee8a674131860ef17898c65308 consensus state has no leader
I20260810 23:32:56.824893 13008 tablet_service.cc:1467] Processing CreateTablet for tablet 74c86b5021fc48879fe60aa92b95243d (DEFAULT_TABLE table=default.Table_4f9225f25cc842fbad1e9ce8c1dba105 [id=891bfbcb1df049deb88545fcc3fb6792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.824764 13514 raft_consensus.cc:348] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.825553 13514 raft_consensus.cc:374] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:56.826565 13008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c86b5021fc48879fe60aa92b95243d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.828060 13514 raft_consensus.cc:729] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:56.792737 14985 raft_consensus.cc:504] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.831851 12757 tablet_service.cc:1467] Processing CreateTablet for tablet e17e5ec604044047b499312e27ca3de4 (DEFAULT_TABLE table=default.Table_5a0a5857ffc649eb96a5e9e271b8ce7a [id=b27a53844fa3419d80a7313ba2437c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.829124 13514 consensus_queue.cc:260] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:56.834584 12763 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d96bc27a87a41908f3ec838c228030d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:55 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:56.837731 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17e5ec604044047b499312e27ca3de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.838804 14985 leader_election.cc:288] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:56.847702 13020 tablet_service.cc:1467] Processing CreateTablet for tablet e17e5ec604044047b499312e27ca3de4 (DEFAULT_TABLE table=default.Table_5a0a5857ffc649eb96a5e9e271b8ce7a [id=b27a53844fa3419d80a7313ba2437c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.831739 12755 tablet_service.cc:1467] Processing CreateTablet for tablet 74c86b5021fc48879fe60aa92b95243d (DEFAULT_TABLE table=default.Table_4f9225f25cc842fbad1e9ce8c1dba105 [id=891bfbcb1df049deb88545fcc3fb6792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.857162 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c86b5021fc48879fe60aa92b95243d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.864078 15011 consensus_queue.cc:1035] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:56.865753 14931 consensus_queue.cc:1035] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:56.867096 12235 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:56.873072 15011 consensus_queue.cc:1035] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:56.877161 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1f24e226b94ccf8ad5a656c80a5b3b" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:56.879813 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17e5ec604044047b499312e27ca3de4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:56.887580 12700 leader_election.cc:341] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:56.820169 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 74c86b5021fc48879fe60aa92b95243d (DEFAULT_TABLE table=default.Table_4f9225f25cc842fbad1e9ce8c1dba105 [id=891bfbcb1df049deb88545fcc3fb6792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.906320 13023 tablet_service.cc:1467] Processing CreateTablet for tablet 6596f813c59943d4a78bd813cab7aa8d (DEFAULT_TABLE table=default.Table_764f15b83d234a90886ebfae787847ad [id=985f8cd96a404f6d8d322ed7f10f968b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.907015 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c86b5021fc48879fe60aa92b95243d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.907665 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6596f813c59943d4a78bd813cab7aa8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:56.821955 13212 leader_election.cc:302] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:56.840817 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1f24e226b94ccf8ad5a656c80a5b3b" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:56.882941 12235 catalog_manager.cc:5495] T 3fef127263f1470ea1056a454971e465 P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f26517150f1c480a877f15aec71d020b (127.11.226.130). New cstate: current_term: 1 leader_uuid: "f26517150f1c480a877f15aec71d020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } }
I20260810 23:32:56.926641 14977 raft_consensus.cc:2781] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:56.927191 14977 raft_consensus.cc:686] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:56.943578 13974 ts_tablet_manager.cc:1918] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260810 23:32:56.946758 12698 leader_election.cc:341] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:56.947253 12698 leader_election.cc:302] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:56.950174 13514 ts_tablet_manager.cc:1428] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.158s user 0.008s sys 0.000s
I20260810 23:32:56.955153 13665 tablet_bootstrap.cc:492] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:56.955825 13665 ts_tablet_manager.cc:1397] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.305s user 0.017s sys 0.005s
I20260810 23:32:56.975450 14121 ts_tablet_manager.cc:1918] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:56.977043 13974 log.cc:1196] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/440de7d9de8d4be9a5322e1fe3388b9b
I20260810 23:32:56.977185 13514 tablet_bootstrap.cc:492] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:56.979504 12228 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:56.981993 12228 catalog_manager.cc:5495] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b reported cstate change: term changed from 0 to 1, leader changed from <none> to f26517150f1c480a877f15aec71d020b (127.11.226.130). New cstate: current_term: 1 leader_uuid: "f26517150f1c480a877f15aec71d020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:56.974433 12794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47b8b57d5b44a018bc2d7b3f976d803" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:56.984579 12794 raft_consensus.cc:3037] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:56.995930 14977 consensus_queue.cc:237] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.004163 13014 tablet_service.cc:1467] Processing CreateTablet for tablet 0703cf71490c4c71a95f55863583a03c (DEFAULT_TABLE table=default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.005641 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0703cf71490c4c71a95f55863583a03c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:57.008281 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 186419us
I20260810 23:32:56.977994 13974 ts_tablet_manager.cc:1939] T 440de7d9de8d4be9a5322e1fe3388b9b P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:57.012814 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 6596f813c59943d4a78bd813cab7aa8d (DEFAULT_TABLE table=default.Table_764f15b83d234a90886ebfae787847ad [id=985f8cd96a404f6d8d322ed7f10f968b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.013932 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6596f813c59943d4a78bd813cab7aa8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.014487 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 230) took 4515 ms. Trace:
I20260810 23:32:57.014801 13974 rpcz_store.cc:276] 0810 23:32:52.499089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.499255 (+ 166us) service_pool.cc:227] Handling call
0810 23:32:56.477377 (+3978122us) ts_tablet_manager.cc:1145] Deleting tablet 440de7d9de8d4be9a5322e1fe3388b9b
0810 23:32:56.477460 (+ 83us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:56.788232 (+310772us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:56.943268 (+155036us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:57.014417 (+ 71149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9501,"tablet-delete.queue_time_us":3922855}
I20260810 23:32:57.011713 13278 tablet_service.cc:1467] Processing CreateTablet for tablet e17e5ec604044047b499312e27ca3de4 (DEFAULT_TABLE table=default.Table_5a0a5857ffc649eb96a5e9e271b8ce7a [id=b27a53844fa3419d80a7313ba2437c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:56.987138 14985 raft_consensus.cc:2726] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:57.026960 15093 raft_consensus.cc:482] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:57.028774 15093 raft_consensus.cc:504] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.034426 13691 tablet_bootstrap.cc:492] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:57.034967 13691 ts_tablet_manager.cc:1397] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.454s user 0.021s sys 0.005s
W20260810 23:32:57.037796 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 140407us
I20260810 23:32:57.045171 13691 raft_consensus.cc:348] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.045890 13691 raft_consensus.cc:374] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:57.046136 13691 raft_consensus.cc:729] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:57.053179 13665 raft_consensus.cc:348] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.055315 12794 raft_consensus.cc:2454] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:57.054319 13691 consensus_queue.cc:260] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.061843 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 6596f813c59943d4a78bd813cab7aa8d (DEFAULT_TABLE table=default.Table_764f15b83d234a90886ebfae787847ad [id=985f8cd96a404f6d8d322ed7f10f968b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.062002 12764 tablet_service.cc:1467] Processing CreateTablet for tablet 0703cf71490c4c71a95f55863583a03c (DEFAULT_TABLE table=default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.063153 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6596f813c59943d4a78bd813cab7aa8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.068368 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0703cf71490c4c71a95f55863583a03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.043915 13514 tablet_bootstrap.cc:654] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:57.111637 13665 raft_consensus.cc:374] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:57.054641 15093 leader_election.cc:288] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:57.082484 13974 tablet_replica.cc:330] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:57.094153 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17e5ec604044047b499312e27ca3de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.113636 13691 ts_tablet_manager.cc:1428] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.078s user 0.008s sys 0.000s
I20260810 23:32:57.116523 13008 tablet_service.cc:1467] Processing CreateTablet for tablet 8258258b2ee040f297512bb5667b6b1b (DEFAULT_TABLE table=default.Table_b138f739a5db4c59af5e8e17ae77a665 [id=abbede238d2641f396da52ac62b64f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.126587 13974 raft_consensus.cc:2229] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 23:32:57.059866 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 5d96bc27a87a41908f3ec838c228030d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d96bc27a87a41908f3ec838c228030d already in progress: creating tablet
W20260810 23:32:57.132596 12217 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 72793.3us
I20260810 23:32:57.132990 13974 pending_rounds.cc:62] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3: Trying to abort 1 pending ops.
I20260810 23:32:57.133226 13974 pending_rounds.cc:68] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317113930038153216 op_type: NO_OP noop_request { }
I20260810 23:32:57.133669 13974 raft_consensus.cc:2866] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:32:57.134028 13974 raft_consensus.cc:2258] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:57.097664 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 440de7d9de8d4be9a5322e1fe3388b9b (table default.Table_3b251f8589be43f29704b8906d49f6cd [id=c56022cc8cad4c7cbefafad0782cefde]) successfully deleted
I20260810 23:32:57.112015 13665 raft_consensus.cc:729] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:56.975919 14121 log.cc:1196] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/133654008f5247b2ac87dcd09195f197
I20260810 23:32:57.140925 13665 consensus_queue.cc:260] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.151336 13665 ts_tablet_manager.cc:1428] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.195s user 0.006s sys 0.000s
I20260810 23:32:57.152467 13665 tablet_bootstrap.cc:492] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:57.153131 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 8258258b2ee040f297512bb5667b6b1b (DEFAULT_TABLE table=default.Table_b138f739a5db4c59af5e8e17ae77a665 [id=abbede238d2641f396da52ac62b64f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.154307 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8258258b2ee040f297512bb5667b6b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.158293 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0136a1e890436fae572b46e64750e7" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:57.160311 13008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8258258b2ee040f297512bb5667b6b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.162918 13974 ts_tablet_manager.cc:1905] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:57.163368 13691 tablet_bootstrap.cc:492] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:57.164170 14121 ts_tablet_manager.cc:1939] T 133654008f5247b2ac87dcd09195f197 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:57.166338 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 0703cf71490c4c71a95f55863583a03c (DEFAULT_TABLE table=default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.167531 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0703cf71490c4c71a95f55863583a03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.170331 12785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0136a1e890436fae572b46e64750e7" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:57.171080 12785 raft_consensus.cc:2454] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:57.180557 13213 leader_election.cc:302] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:57.181969 15087 raft_consensus.cc:2781] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:57.182389 15087 raft_consensus.cc:482] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:57.182786 15087 raft_consensus.cc:3037] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:57.191934 14931 consensus_queue.cc:1035] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:57.194299 15093 consensus_queue.cc:1035] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:57.198284 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 8258258b2ee040f297512bb5667b6b1b (DEFAULT_TABLE table=default.Table_b138f739a5db4c59af5e8e17ae77a665 [id=abbede238d2641f396da52ac62b64f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.199611 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8258258b2ee040f297512bb5667b6b1b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:57.198319 13212 leader_election.cc:341] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:57.219486 13020 tablet_service.cc:1467] Processing CreateTablet for tablet ff732fd231a74846b12686329ffe53c3 (DEFAULT_TABLE table=default.Table_fc0d875790454fcba23aa9c7e2ee1c4e [id=225155f28cad4653997e017cdd21008a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.221477 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff732fd231a74846b12686329ffe53c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.234273 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:57.238447 12236 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e] alter complete (version 1)
W20260810 23:32:57.241981 12236 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 31) took 4511 ms (client timeout 1000 ms). Trace:
I20260810 23:32:57.243714 13665 tablet_bootstrap.cc:654] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:57.263561 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 65272.1us
I20260810 23:32:57.267812 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet 42517f11422b42c39db282e2c5f584a3 with delete_type TABLET_DATA_DELETED (replaced by tablet a480093c481941f0bb35949c7606f37f) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:57.268720 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c29b11192604ac18c21b7bdc0238fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:38 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:57.261197 13014 tablet_service.cc:1467] Processing CreateTablet for tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2 (DEFAULT_TABLE table=default.Table_68bcc5970e834ef8bb810ac4f663c1d0 [id=956f3847e6384c07a3c61ee05e5ff9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.270285 13278 tablet_service.cc:1514] Processing DeleteTablet for tablet b261673faa964b65af3576c89e302864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:57.256736 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 208) took 6909 ms. Trace:
I20260810 23:32:57.270638 14121 rpcz_store.cc:276] 0810 23:32:50.347618 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.347926 (+ 308us) service_pool.cc:227] Handling call
0810 23:32:56.520081 (+6172155us) ts_tablet_manager.cc:1145] Deleting tablet 133654008f5247b2ac87dcd09195f197
0810 23:32:56.537980 (+ 17899us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:56.892471 (+354491us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:56.975132 (+ 82661us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:57.256654 (+281522us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6170996}
I20260810 23:32:57.271275 13283 tablet_service.cc:1467] Processing CreateTablet for tablet ff732fd231a74846b12686329ffe53c3 (DEFAULT_TABLE table=default.Table_fc0d875790454fcba23aa9c7e2ee1c4e [id=225155f28cad4653997e017cdd21008a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.272045 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 133654008f5247b2ac87dcd09195f197 (table default.Table_d664d07d116e4f2aa4fc263449eb8dd0 [id=04ed6428e4df49b2ab0438035da70731]) successfully deleted
I20260810 23:32:57.274173 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff732fd231a74846b12686329ffe53c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.198119 15087 raft_consensus.cc:504] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.283983 12226 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_7221fbab5f0b4fa892342870790a2f7e [id=1ea3dfab929845f89d4680129ae45d94] alter complete (version 2)
I20260810 23:32:57.285192 15087 leader_election.cc:288] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:57.287603 13017 tablet_service.cc:1467] Processing CreateTablet for tablet f47d8e462d814a15a82fd0d5218e2d0e (DEFAULT_TABLE table=default.Table_95b3b58c2c4948c199a9b573367fa598 [id=56bede34f9a1429daaed5c6077be5513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.303874 13665 tablet_bootstrap.cc:492] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
W20260810 23:32:57.242403 12236 rpcz_store.cc:269] 0810 23:32:52.730798 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.745977 (+ 15179us) service_pool.cc:227] Handling call
0810 23:32:57.241872 (+4495895us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:53.742903 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:53.743661 (+ 758us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:53.743699 (+ 38us) write_op.cc:435] Acquired schema lock
0810 23:32:53.743714 (+ 15us) tablet.cc:660] Decoding operations
0810 23:32:53.816187 (+ 72473us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:53.816245 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:53.816270 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:53.816639 (+ 369us) tablet.cc:699] Row locks acquired
0810 23:32:53.816659 (+ 20us) write_op.cc:260] PREPARE: finished
0810 23:32:53.816840 (+ 181us) write_op.cc:270] Start()
0810 23:32:53.816952 (+ 112us) write_op.cc:276] Timestamp: P: 1786404773816744 usec, L: 0
0810 23:32:53.817061 (+ 109us) op_driver.cc:331] REPLICATION: starting
0810 23:32:53.817877 (+ 816us) log.cc:842] Serialized 356 byte log entry
0810 23:32:53.833057 (+ 15180us) op_driver.cc:444] REPLICATION: finished
0810 23:32:53.834657 (+ 1600us) write_op.cc:301] APPLY: starting
0810 23:32:53.834728 (+ 71us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:53.834832 (+ 104us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:53.834951 (+ 119us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:53.835171 (+ 220us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:53.835315 (+ 144us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:53.835360 (+ 45us) write_op.cc:312] APPLY: finished
0810 23:32:53.835659 (+ 299us) log.cc:842] Serialized 23 byte log entry
0810 23:32:53.836579 (+ 920us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:53.836661 (+ 82us) write_op.cc:454] Released schema lock
0810 23:32:53.836698 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:53.837090 (+ 392us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:53.837115 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":1416,"apply.run_cpu_time_us":1725,"apply.run_wall_time_us":2647,"num_ops":1,"prepare.queue_time_us":589605,"prepare.run_cpu_time_us":3554,"prepare.run_wall_time_us":258536,"raft.queue_time_us":125,"raft.run_cpu_time_us":967,"raft.run_wall_time_us":14372,"replication_time_us":15898}]]}
W20260810 23:32:57.309371 12236 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 40) took 1021 ms (client timeout 1000 ms). Trace:
I20260810 23:32:57.311983 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 7c10d464bb05463f99541ca1ea09d5dd (DEFAULT_TABLE table=default.Table_c244137f10db487f95395a64261b74b1 [id=12051526e7a441649a43d4edbd47a578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.313252 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.309121 13278 tablet_service.cc:1467] Processing CreateTablet for tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2 (DEFAULT_TABLE table=default.Table_68bcc5970e834ef8bb810ac4f663c1d0 [id=956f3847e6384c07a3c61ee05e5ff9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.311029 13282 tablet_service.cc:1467] Processing CreateTablet for tablet f47d8e462d814a15a82fd0d5218e2d0e (DEFAULT_TABLE table=default.Table_95b3b58c2c4948c199a9b573367fa598 [id=56bede34f9a1429daaed5c6077be5513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.314185 13665 ts_tablet_manager.cc:1397] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.162s user 0.013s sys 0.003s
I20260810 23:32:57.314657 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47d8e462d814a15a82fd0d5218e2d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.318704 13279 tablet_service.cc:1467] Processing CreateTablet for tablet 7c10d464bb05463f99541ca1ea09d5dd (DEFAULT_TABLE table=default.Table_c244137f10db487f95395a64261b74b1 [id=12051526e7a441649a43d4edbd47a578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.319804 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c10d464bb05463f99541ca1ea09d5dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:57.320950 12236 rpcz_store.cc:269] 0810 23:32:56.287948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.308928 (+1020980us) service_pool.cc:214] Skipping call since client already timed out
0810 23:32:57.309278 (+ 350us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260810 23:32:57.325773 12240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 92b81e7056a441d2854ba5ab6b4c5433
I20260810 23:32:57.327479 12240 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 315) took 2769 ms. Trace:
W20260810 23:32:57.325898 12224 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c01ede4b52c3498a924340b39c3c2435 consensus state has no leader
I20260810 23:32:57.328562 12224 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 314) took 2997 ms. Trace:
I20260810 23:32:57.328871 12224 rpcz_store.cc:276] 0810 23:32:54.331355 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:55.555914 (+1224559us) service_pool.cc:227] Handling call
0810 23:32:55.559495 (+ 3581us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e77d6 00007f0ce6e2d634 00007f0ce6e2d5c2 00007f0ce6e2d4f1 00007f0ce6e2d46c 00007f0ce6e2c722 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:55.562245 (+ 2750us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:55.563252 (+ 1007us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:55.564152 (+ 900us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:55.564435 (+ 283us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:55.730412 (+165977us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:57.328472 (+1598060us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:55.620409 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:55.620853 (+ 444us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:55.620878 (+ 25us) write_op.cc:435] Acquired schema lock
0810 23:32:55.620891 (+ 13us) tablet.cc:660] Decoding operations
0810 23:32:55.621222 (+ 331us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:55.621272 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:55.621298 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:55.621498 (+ 200us) tablet.cc:699] Row locks acquired
0810 23:32:55.621513 (+ 15us) write_op.cc:260] PREPARE: finished
0810 23:32:55.621655 (+ 142us) write_op.cc:270] Start()
0810 23:32:55.621757 (+ 102us) write_op.cc:276] Timestamp: P: 1786404775621553 usec, L: 0
0810 23:32:55.621779 (+ 22us) op_driver.cc:331] REPLICATION: starting
0810 23:32:55.622434 (+ 655us) log.cc:842] Serialized 405 byte log entry
0810 23:32:55.726954 (+104520us) op_driver.cc:444] REPLICATION: finished
0810 23:32:55.727369 (+ 415us) write_op.cc:301] APPLY: starting
0810 23:32:55.727479 (+ 110us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:55.727646 (+ 167us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:55.727823 (+ 177us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:55.728061 (+ 238us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:55.728213 (+ 152us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:55.728262 (+ 49us) write_op.cc:312] APPLY: finished
0810 23:32:55.728504 (+ 242us) log.cc:842] Serialized 23 byte log entry
0810 23:32:55.729360 (+ 856us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:55.729459 (+ 99us) write_op.cc:454] Released schema lock
0810 23:32:55.729516 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:55.729770 (+ 254us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:55.729804 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":5177600,"child_traces":[["op",{"apply.queue_time_us":224,"apply.run_cpu_time_us":1990,"apply.run_wall_time_us":2639,"num_ops":1,"prepare.queue_time_us":177,"prepare.run_cpu_time_us":2375,"prepare.run_wall_time_us":2374,"raft.queue_time_us":53517,"raft.run_cpu_time_us":1073,"raft.run_wall_time_us":1944,"replication_time_us":105048}]]}
I20260810 23:32:57.332557 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.338160 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b81e7056a441d2854ba5ab6b4c5433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:55 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:57.339891 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0136a1e890436fae572b46e64750e7" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:57.345995 13031 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb73a8ea7e44be9aad20a5548ef81cc (DEFAULT_TABLE table=default.Table_98f0ace30b644009a7f4cd8cbe85217c [id=9c3ca59291864ec5ae5bc902a81c2110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.347245 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb73a8ea7e44be9aad20a5548ef81cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.327831 12240 rpcz_store.cc:276] 0810 23:32:54.558400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:55.580388 (+1021988us) service_pool.cc:227] Handling call
0810 23:32:55.668727 (+ 88339us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:55.718687 (+ 49960us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:55.718919 (+ 232us) catalog_manager.cc:2761] Locking tablets
0810 23:32:56.587741 (+868822us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:56.814239 (+226498us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:56.814547 (+ 308us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:57.325624 (+511077us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:57.327384 (+ 1760us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:56.590551 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:56.591015 (+ 464us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:56.591044 (+ 29us) write_op.cc:435] Acquired schema lock
0810 23:32:56.591061 (+ 17us) tablet.cc:660] Decoding operations
0810 23:32:56.591445 (+ 384us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:56.591491 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:56.591514 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:56.591834 (+ 320us) tablet.cc:699] Row locks acquired
0810 23:32:56.591851 (+ 17us) write_op.cc:260] PREPARE: finished
0810 23:32:56.592007 (+ 156us) write_op.cc:270] Start()
0810 23:32:56.592183 (+ 176us) write_op.cc:276] Timestamp: P: 1786404776591954 usec, L: 0
0810 23:32:56.592214 (+ 31us) op_driver.cc:331] REPLICATION: starting
0810 23:32:56.656758 (+ 64544us) log.cc:842] Serialized 765 byte log entry
0810 23:32:56.658845 (+ 2087us) spinlock_profiling.cc:243] Waited 63.6 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:56.738303 (+ 79458us) op_driver.cc:444] REPLICATION: finished
0810 23:32:56.812369 (+ 74066us) write_op.cc:301] APPLY: starting
0810 23:32:56.812431 (+ 62us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:56.812556 (+ 125us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:56.812577 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:56.812973 (+ 396us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:56.813149 (+ 176us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:56.813206 (+ 57us) write_op.cc:312] APPLY: finished
0810 23:32:56.813446 (+ 240us) log.cc:842] Serialized 29 byte log entry
0810 23:32:56.813634 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:56.813719 (+ 85us) write_op.cc:454] Released schema lock
0810 23:32:56.813754 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:56.813984 (+ 230us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:56.814007 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":35086,"apply.run_cpu_time_us":1898,"apply.run_wall_time_us":1925,"num_ops":2,"prepare.queue_time_us":282,"prepare.run_cpu_time_us":4616,"prepare.run_wall_time_us":78714,"raft.queue_time_us":269,"raft.run_cpu_time_us":879,"raft.run_wall_time_us":879,"replication_time_us":132344,"spinlock_wait_cycles":146358144}]]}
I20260810 23:32:57.353199 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 210) took 6548 ms. Trace:
I20260810 23:32:57.353650 14121 rpcz_store.cc:276] 0810 23:32:50.804574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:50.804809 (+ 235us) service_pool.cc:227] Handling call
0810 23:32:57.272156 (+6467347us) ts_tablet_manager.cc:1145] Deleting tablet 0be35dabbec44caebd1df687b167f34a
0810 23:32:57.315998 (+ 43842us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:57.353104 (+ 37106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6466378}
I20260810 23:32:57.356144 12240 master_service.cc:946] ReplaceTablet: received request to replace tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:57.358810 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c10d464bb05463f99541ca1ea09d5dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:57.359329 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 0be35dabbec44caebd1df687b167f34a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0be35dabbec44caebd1df687b167f34a already in progress: creating tablet
I20260810 23:32:57.364492 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 217) took 5190 ms. Trace:
I20260810 23:32:57.364850 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b81e7056a441d2854ba5ab6b4c5433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:55 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:57.364951 14121 rpcz_store.cc:276] 0810 23:32:52.174617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.179950 (+ 5333us) service_pool.cc:227] Handling call
0810 23:32:57.359420 (+5179470us) ts_tablet_manager.cc:1145] Deleting tablet 4d1d3bd06a9a4627a1a7e83b22dba2af
0810 23:32:57.360985 (+ 1565us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:57.364416 (+ 3431us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5178566}
W20260810 23:32:57.366124 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 4d1d3bd06a9a4627a1a7e83b22dba2af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d1d3bd06a9a4627a1a7e83b22dba2af already in progress: creating tablet
I20260810 23:32:57.366560 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 216) took 5201 ms. Trace:
I20260810 23:32:57.366549 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb73a8ea7e44be9aad20a5548ef81cc (DEFAULT_TABLE table=default.Table_98f0ace30b644009a7f4cd8cbe85217c [id=9c3ca59291864ec5ae5bc902a81c2110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.366899 14121 rpcz_store.cc:276] 0810 23:32:52.165096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.165590 (+ 494us) service_pool.cc:227] Handling call
0810 23:32:57.366218 (+5200628us) ts_tablet_manager.cc:1145] Deleting tablet 1b1f24e226b94ccf8ad5a656c80a5b3b
0810 23:32:57.366258 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:57.366507 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5181642}
I20260810 23:32:57.367687 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb73a8ea7e44be9aad20a5548ef81cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:57.367987 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b1f24e226b94ccf8ad5a656c80a5b3b already in progress: creating tablet
I20260810 23:32:57.376271 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47d8e462d814a15a82fd0d5218e2d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.382439 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2 (DEFAULT_TABLE table=default.Table_68bcc5970e834ef8bb810ac4f663c1d0 [id=956f3847e6384c07a3c61ee05e5ff9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.383648 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.384327 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 218) took 5208 ms. Trace:
I20260810 23:32:57.384704 14121 rpcz_store.cc:276] 0810 23:32:52.175832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.181153 (+ 5321us) service_pool.cc:227] Handling call
0810 23:32:57.367894 (+5186741us) ts_tablet_manager.cc:1145] Deleting tablet a480093c481941f0bb35949c7606f37f
0810 23:32:57.367931 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:57.384240 (+ 16309us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5181511}
I20260810 23:32:57.385237 13974 ts_tablet_manager.cc:1918] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:57.385684 13974 log.cc:1196] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/4d697e7533af4fc0a15cb41f3fcce4db
I20260810 23:32:57.386152 14121 tablet_replica.cc:330] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:57.386504 13974 ts_tablet_manager.cc:1939] T 4d697e7533af4fc0a15cb41f3fcce4db P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:57.386945 14121 raft_consensus.cc:2229] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:57.386947 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0136a1e890436fae572b46e64750e7" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:57.387549 12806 raft_consensus.cc:3037] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:57.391546 13212 leader_election.cc:341] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:57.390806 13665 raft_consensus.cc:348] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:57.395979 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet a480093c481941f0bb35949c7606f37f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a480093c481941f0bb35949c7606f37f already in progress: creating tablet
I20260810 23:32:57.396405 13665 raft_consensus.cc:374] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:57.396695 13665 raft_consensus.cc:729] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:57.397404 13665 consensus_queue.cc:260] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.399041 12755 tablet_service.cc:1467] Processing CreateTablet for tablet ff732fd231a74846b12686329ffe53c3 (DEFAULT_TABLE table=default.Table_fc0d875790454fcba23aa9c7e2ee1c4e [id=225155f28cad4653997e017cdd21008a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.399495 13665 ts_tablet_manager.cc:1428] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.085s user 0.002s sys 0.002s
I20260810 23:32:57.405584 15087 consensus_queue.cc:1035] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:57.406630 13665 tablet_bootstrap.cc:492] T cd24471dd52149d395808493ad9fa68c P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:57.387995 14121 raft_consensus.cc:2258] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:57.401751 12763 tablet_service.cc:1467] Processing CreateTablet for tablet f47d8e462d814a15a82fd0d5218e2d0e (DEFAULT_TABLE table=default.Table_95b3b58c2c4948c199a9b573367fa598 [id=56bede34f9a1429daaed5c6077be5513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.422345 14121 ts_tablet_manager.cc:1905] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:57.428512 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff732fd231a74846b12686329ffe53c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.433264 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47d8e462d814a15a82fd0d5218e2d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.437783 12766 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb73a8ea7e44be9aad20a5548ef81cc (DEFAULT_TABLE table=default.Table_98f0ace30b644009a7f4cd8cbe85217c [id=9c3ca59291864ec5ae5bc902a81c2110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.438985 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b81e7056a441d2854ba5ab6b4c5433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:55 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:57.443071 12224 catalog_manager.cc:5495] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f26517150f1c480a877f15aec71d020b (127.11.226.130). New cstate: current_term: 1 leader_uuid: "f26517150f1c480a877f15aec71d020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } }
I20260810 23:32:57.445752 12224 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:57.446539 12224 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet c01ede4b52c3498a924340b39c3c2435 (table default.Table_ebe848b2f9ba4077a556369c87aa436a [id=ae7b5faec4c34d35981e0a1d464202ef]). Expected version 1 got 0
W20260810 23:32:57.449957 12952 consensus_peers.cc:487] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d0d9227d8574de8bd73041f3e8043a0. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:57.450722 12224 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]). Expected version 1 got 0
I20260810 23:32:57.457950 13514 tablet_bootstrap.cc:492] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:57.458529 13514 ts_tablet_manager.cc:1397] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.486s user 0.028s sys 0.004s
I20260810 23:32:57.462011 13514 raft_consensus.cc:348] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.462846 13514 raft_consensus.cc:374] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:57.463369 13514 raft_consensus.cc:729] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:57.471148 15082 tablet_replica.cc:330] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:57.471426 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb73a8ea7e44be9aad20a5548ef81cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.473300 15082 raft_consensus.cc:2229] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:57.474094 15082 raft_consensus.cc:2258] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:57.477131 15082 ts_tablet_manager.cc:1905] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:57.506572 15087 raft_consensus.cc:482] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:57.507331 15087 raft_consensus.cc:504] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.509174 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 231) took 4862 ms. Trace:
I20260810 23:32:57.509598 13974 rpcz_store.cc:276] 0810 23:32:52.646539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.646830 (+ 291us) service_pool.cc:227] Handling call
0810 23:32:57.015926 (+4369096us) ts_tablet_manager.cc:1145] Deleting tablet 4d697e7533af4fc0a15cb41f3fcce4db
0810 23:32:57.015984 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:57.246854 (+230870us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:57.384949 (+138095us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:57.509085 (+124136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4368059}
I20260810 23:32:57.512300 15087 leader_election.cc:288] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:57.512228 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 4d697e7533af4fc0a15cb41f3fcce4db (table default.Table_26bf2c7d86704b5b981ecebb4e90ba75 [id=cefe7b97c1db42ef8712bf3c7a359af2]) successfully deleted
I20260810 23:32:57.512781 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 241) took 3163 ms. Trace:
I20260810 23:32:57.513289 13974 rpcz_store.cc:276] 0810 23:32:54.349187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:54.349508 (+ 321us) service_pool.cc:227] Handling call
0810 23:32:57.512295 (+3162787us) ts_tablet_manager.cc:1145] Deleting tablet 0df95e01ce894651a0f49396baf49c51
0810 23:32:57.512353 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:57.512711 (+ 358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3151330}
I20260810 23:32:57.470091 12764 tablet_service.cc:1467] Processing CreateTablet for tablet 7c10d464bb05463f99541ca1ea09d5dd (DEFAULT_TABLE table=default.Table_c244137f10db487f95395a64261b74b1 [id=12051526e7a441649a43d4edbd47a578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:57.529003 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 133654008f5247b2ac87dcd09195f197 with delete_type TABLET_DATA_DELETED (replaced by tablet ca8c0d1dacc344cbacb0c5988f0e2221) from {username='slave'} at 127.0.0.1:58598
W20260810 23:32:57.539597 12235 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 60) took 3083 ms (client timeout 1000 ms). Trace:
W20260810 23:32:57.541819 12235 rpcz_store.cc:269] 0810 23:32:54.455985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:54.879188 (+423203us) service_pool.cc:227] Handling call
0810 23:32:56.966322 (+2087134us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:56.966354 (+ 32us) write_op.cc:454] Released schema lock
0810 23:32:57.539495 (+573141us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:56.942119 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:56.942769 (+ 650us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:56.942812 (+ 43us) write_op.cc:435] Acquired schema lock
0810 23:32:56.942837 (+ 25us) tablet.cc:660] Decoding operations
0810 23:32:56.943269 (+ 432us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:56.943327 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:56.943358 (+ 31us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:56.943557 (+ 199us) tablet.cc:699] Row locks acquired
0810 23:32:56.943578 (+ 21us) write_op.cc:260] PREPARE: finished
0810 23:32:56.943745 (+ 167us) write_op.cc:270] Start()
0810 23:32:56.943847 (+ 102us) write_op.cc:276] Timestamp: P: 1786404776943698 usec, L: 0
0810 23:32:56.943870 (+ 23us) op_driver.cc:331] REPLICATION: starting
0810 23:32:56.944952 (+ 1082us) log.cc:842] Serialized 536 byte log entry
0810 23:32:56.963023 (+ 18071us) op_driver.cc:444] REPLICATION: finished
0810 23:32:56.964279 (+ 1256us) write_op.cc:301] APPLY: starting
0810 23:32:56.964364 (+ 85us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:56.964501 (+ 137us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:56.964628 (+ 127us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:56.964904 (+ 276us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:56.965104 (+ 200us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:56.965171 (+ 67us) write_op.cc:312] APPLY: finished
0810 23:32:56.965422 (+ 251us) log.cc:842] Serialized 23 byte log entry
0810 23:32:56.965624 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:56.965719 (+ 95us) write_op.cc:454] Released schema lock
0810 23:32:56.965772 (+ 53us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":14,"child_traces":[["op",{"apply.queue_time_us":988,"apply.run_cpu_time_us":1798,"apply.run_wall_time_us":1798,"num_ops":1,"prepare.queue_time_us":244,"prepare.run_cpu_time_us":3293,"prepare.run_wall_time_us":4699,"raft.queue_time_us":252,"raft.run_cpu_time_us":1020,"raft.run_wall_time_us":1755,"replication_time_us":19032}]]}
I20260810 23:32:57.548107 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet fd9d90e1ceab443dbec12b2c81b975e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:57.548838 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0dd19112504f8ca57f64dcf0bd50a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:57.550107 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdec13ee70d402fa3a08119e7bf1313 with delete_type TABLET_DATA_DELETED (replaced by tablet d603569c54ef4ffba9941d0cc3e0f97f) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:57.551222 12760 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d0d9227d8574de8bd73041f3e8043a0 with delete_type TABLET_DATA_DELETED (replaced by tablet fb8d482c6ee54e45a41201d98a5b60b4) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:57.557034 15091 raft_consensus.cc:482] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:57.557624 15091 raft_consensus.cc:504] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.560604 15091 leader_election.cc:288] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:32:57.553768 12229 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 59) took 4117 ms (client timeout 1000 ms). Trace:
W20260810 23:32:57.562141 12229 rpcz_store.cc:269] 0810 23:32:53.436478 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:53.474176 (+ 37698us) service_pool.cc:227] Handling call
0810 23:32:57.553663 (+4079487us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":11}
I20260810 23:32:57.563891 12785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bce222cd5ab41fdb12fff3fe3dfbd3c" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:57.565070 12785 raft_consensus.cc:2454] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:57.570163 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c10d464bb05463f99541ca1ea09d5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:57.576709 12235 master_service.cc:946] ReplaceTablet: received request to replace tablet 62705f99687340aaa1aca0942b28487f from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:57.578260 12235 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 318) took 2278 ms. Trace:
I20260810 23:32:57.578660 12235 rpcz_store.cc:276] 0810 23:32:55.303404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.576513 (+2273109us) service_pool.cc:227] Handling call
0810 23:32:57.578129 (+ 1616us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 23:32:57.554309 12223 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 33) took 2152 ms (client timeout 1000 ms). Trace:
I20260810 23:32:57.584671 12226 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_d8a0e3e655b9417d9df3ff6d82b76f15 [id=d0b86eafe9544cd98d8fce8ece3b5c10] alter complete (version 1)
I20260810 23:32:57.590036 13514 consensus_queue.cc:260] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.592847 13514 ts_tablet_manager.cc:1428] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.134s user 0.007s sys 0.001s
I20260810 23:32:57.593962 13514 tablet_bootstrap.cc:492] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:57.567054 13213 leader_election.cc:302] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:57.601303 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 242) took 3231 ms. Trace:
I20260810 23:32:57.601883 13974 rpcz_store.cc:276] 0810 23:32:54.370152 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:54.373042 (+ 2890us) service_pool.cc:227] Handling call
0810 23:32:57.600825 (+3227783us) ts_tablet_manager.cc:1145] Deleting tablet f0cba350e63e46a894a89ccc5727aab8
0810 23:32:57.600880 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:57.601231 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3224115}
I20260810 23:32:57.603547 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 243) took 2757 ms. Trace:
I20260810 23:32:57.604590 12806 raft_consensus.cc:2454] T 0a0136a1e890436fae572b46e64750e7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
W20260810 23:32:57.584095 12223 rpcz_store.cc:269] 0810 23:32:55.402309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:55.527159 (+124850us) service_pool.cc:227] Handling call
0810 23:32:55.571672 (+ 44513us) spinlock_profiling.cc:243] Waited 39.2 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:57.554219 (+1982547us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22391,"reported_tablets":17,"spinlock_wait_cycles":90103552}
I20260810 23:32:57.609164 13514 tablet_bootstrap.cc:654] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:57.612406 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_de805fc39d9948e1a6c78d2979ad6e73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:57.614948 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de805fc39d9948e1a6c78d2979ad6e73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:57.623600 12769 tablet_service.cc:1514] Processing DeleteTablet for tablet f47b8b57d5b44a018bc2d7b3f976d803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:57.640470 13514 tablet_bootstrap.cc:492] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:57.641076 13514 ts_tablet_manager.cc:1397] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.002s
I20260810 23:32:57.644400 13514 raft_consensus.cc:348] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:57.645251 13514 raft_consensus.cc:374] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:57.645509 13514 raft_consensus.cc:729] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:57.646134 13514 consensus_queue.cc:260] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:57.661844 12760 tablet_service.cc:1514] Processing DeleteTablet for tablet a41ddf9f6f71443ba75c97d2b010372c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:57.663266 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:51 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:57.663635 13514 ts_tablet_manager.cc:1428] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.022s user 0.002s sys 0.004s
I20260810 23:32:57.668718 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3049bcdc6e43c795961c74f0d008f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:57.672362 12769 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e8311c941b43058384cee6d8263dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:57.710554 13514 tablet_bootstrap.cc:492] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:57.721278 13691 tablet_bootstrap.cc:700] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b: Time spent opening tablet: real 0.350s user 0.001s sys 0.000s
I20260810 23:32:57.722064 13514 tablet_bootstrap.cc:654] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:57.742758 13514 tablet_bootstrap.cc:492] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:57.743271 13514 ts_tablet_manager.cc:1397] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.008s
I20260810 23:32:57.746439 13514 raft_consensus.cc:348] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.747460 13514 raft_consensus.cc:374] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:57.636554 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d697e7533af4fc0a15cb41f3fcce4db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:57.636765 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 69915.2us
I20260810 23:32:57.755173 13974 rpcz_store.cc:276] 0810 23:32:54.846275 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:54.852239 (+ 5964us) service_pool.cc:227] Handling call
0810 23:32:57.603155 (+2750916us) ts_tablet_manager.cc:1145] Deleting tablet 4f453bd72168429086837b2b9b09fc78
0810 23:32:57.603218 (+ 63us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:57.603488 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2749990}
I20260810 23:32:57.747778 13514 raft_consensus.cc:729] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:57.756728 13213 leader_election.cc:302] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:32:57.757009 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 247) took 1078 ms. Trace:
I20260810 23:32:57.756716 13514 consensus_queue.cc:260] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.638717 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d0d9227d8574de8bd73041f3e8043a0 with delete_type TABLET_DATA_DELETED (replaced by tablet fb8d482c6ee54e45a41201d98a5b60b4) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:57.757339 13974 rpcz_store.cc:276] 0810 23:32:56.678673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:56.678794 (+ 121us) service_pool.cc:227] Handling call
0810 23:32:57.756571 (+1077777us) ts_tablet_manager.cc:1145] Deleting tablet 5d96bc27a87a41908f3ec838c228030d
0810 23:32:57.756623 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:57.756949 (+ 326us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1076366}
I20260810 23:32:57.610782 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_f64ee2d577744a919f65f54c28ccc88b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:57.637835 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 0df95e01ce894651a0f49396baf49c51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0df95e01ce894651a0f49396baf49c51 already in progress: creating tablet
I20260810 23:32:57.759171 13514 ts_tablet_manager.cc:1428] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.016s user 0.003s sys 0.004s
W20260810 23:32:57.759479 12214 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 121683us
W20260810 23:32:57.760255 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet f0cba350e63e46a894a89ccc5727aab8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0cba350e63e46a894a89ccc5727aab8 already in progress: creating tablet
I20260810 23:32:57.639256 13276 tablet_service.cc:1514] Processing DeleteTablet for tablet 423e8d583d704fa9834026ebcd57e17e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:57.658519 12224 catalog_manager.cc:5687] Failed to send AlterTable RPC for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 4d1b9d20d09c4d1ba9a4b10959807e50 consensus state has no leader
W20260810 23:32:57.671259 12237 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 61) took 1840 ms (client timeout 1000 ms). Trace:
W20260810 23:32:57.761585 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 4f453bd72168429086837b2b9b09fc78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f453bd72168429086837b2b9b09fc78 already in progress: creating tablet
W20260810 23:32:57.762315 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 5d96bc27a87a41908f3ec838c228030d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d96bc27a87a41908f3ec838c228030d already in progress: creating tablet
W20260810 23:32:57.765281 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f64ee2d577744a919f65f54c28ccc88b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:57.650053 13665 tablet_bootstrap.cc:654] T cd24471dd52149d395808493ad9fa68c P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:57.641167 15087 raft_consensus.cc:2781] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:57.769281 15087 raft_consensus.cc:482] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:57.769704 15087 raft_consensus.cc:3037] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:57.658181 13277 tablet_service.cc:1514] Processing DeleteTablet for tablet fd9d90e1ceab443dbec12b2c81b975e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:57.761330 12237 rpcz_store.cc:269] 0810 23:32:55.831075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:56.695720 (+864645us) service_pool.cc:227] Handling call
0810 23:32:56.926593 (+230873us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:57.671067 (+744474us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":19,"spinlock_wait_cycles":243690880}
I20260810 23:32:57.775028 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_98ae6d61510e4c97bea03912bdbbe8cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:57.777311 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98ae6d61510e4c97bea03912bdbbe8cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:57.778173 12226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 32) took 3797 ms (client timeout 1000 ms). Trace:
W20260810 23:32:57.782172 12234 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 37) took 4751 ms (client timeout 1000 ms). Trace:
I20260810 23:32:57.782497 13691 tablet_bootstrap.cc:654] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:57.782677 12234 rpcz_store.cc:269] 0810 23:32:53.030597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:53.186332 (+155735us) service_pool.cc:227] Handling call
0810 23:32:53.247171 (+ 60839us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:57.782032 (+4534861us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":247324,"reported_tablets":23,"spinlock_wait_cycles":91615616}
I20260810 23:32:57.804042 15011 raft_consensus.cc:482] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:57.804760 15011 raft_consensus.cc:504] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.807400 15011 leader_election.cc:288] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:57.817032 12240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea
I20260810 23:32:57.820688 12240 catalog_manager.cc:6565] ReplaceTablet: tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea deleted and replaced by tablet 7a0e69195596486fb03256baa22446e0
I20260810 23:32:57.821352 12240 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 317) took 2522 ms. Trace:
W20260810 23:32:57.826565 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4d1b9d20d09c4d1ba9a4b10959807e50 consensus state has no leader
I20260810 23:32:57.821667 12240 rpcz_store.cc:276] 0810 23:32:55.299488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.355927 (+2056439us) service_pool.cc:227] Handling call
0810 23:32:57.821262 (+465335us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:57.365147 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:57.365626 (+ 479us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:57.365657 (+ 31us) write_op.cc:435] Acquired schema lock
0810 23:32:57.365672 (+ 15us) tablet.cc:660] Decoding operations
0810 23:32:57.366962 (+ 1290us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:57.367016 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:57.367041 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:57.367290 (+ 249us) tablet.cc:699] Row locks acquired
0810 23:32:57.367302 (+ 12us) write_op.cc:260] PREPARE: finished
0810 23:32:57.367452 (+ 150us) write_op.cc:270] Start()
0810 23:32:57.367595 (+ 143us) write_op.cc:276] Timestamp: P: 1786404777367404 usec, L: 0
0810 23:32:57.367614 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 23:32:57.369078 (+ 1464us) log.cc:842] Serialized 631 byte log entry
0810 23:32:57.380394 (+ 11316us) op_driver.cc:444] REPLICATION: finished
0810 23:32:57.382000 (+ 1606us) write_op.cc:301] APPLY: starting
0810 23:32:57.382067 (+ 67us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:57.382187 (+ 120us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:57.382288 (+ 101us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:57.382699 (+ 411us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:57.382960 (+ 261us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:57.383012 (+ 52us) write_op.cc:312] APPLY: finished
0810 23:32:57.383212 (+ 200us) log.cc:842] Serialized 29 byte log entry
0810 23:32:57.383372 (+ 160us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:57.383455 (+ 83us) write_op.cc:454] Released schema lock
0810 23:32:57.383491 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:57.383683 (+ 192us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:57.383705 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1403,"apply.run_cpu_time_us":1911,"apply.run_wall_time_us":1911,"num_ops":2,"prepare.queue_time_us":2741,"prepare.run_cpu_time_us":2593,"prepare.run_wall_time_us":5500,"raft.queue_time_us":169,"raft.run_cpu_time_us":721,"raft.run_wall_time_us":721,"replication_time_us":12688}]]}
I20260810 23:32:57.861778 14979 raft_consensus.cc:482] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:57.862545 14979 raft_consensus.cc:504] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.867516 14979 leader_election.cc:288] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:57.870806 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 42517f11422b42c39db282e2c5f584a3 with delete_type TABLET_DATA_DELETED (replaced by tablet a480093c481941f0bb35949c7606f37f) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:57.871526 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet 081b4e1f53c74f768a4bf401ae22b585 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:32 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:57.872299 13014 tablet_service.cc:1514] Processing DeleteTablet for tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e with delete_type TABLET_DATA_DELETED (replaced by tablet d97b1c97910e450b8bc42e0c7a970ce0) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:57.872398 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet b261673faa964b65af3576c89e302864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:35 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:57.873684 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4f55ef709d473f89a16d32160eb61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:57.874941 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c45b71432a45269cd6a106fc5f7292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:57.876379 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ad5d63a2104995990a86a000963cdc with delete_type TABLET_DATA_DELETED (replaced by tablet 8fb2ee1c9b41478da001aa67d7a7ad3c) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:57.877911 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet db363eb0f98d460eb6f5a577233d8ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:36 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:57.883054 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea with delete_type TABLET_DATA_DELETED (replaced by tablet 7a0e69195596486fb03256baa22446e0) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:57.884661 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d4c879ca6a402dbafe91940e7b367a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:39 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:57.887672 14121 ts_tablet_manager.cc:1918] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:57.888226 14121 log.cc:1196] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/fd9d90e1ceab443dbec12b2c81b975e8
I20260810 23:32:57.889232 14121 ts_tablet_manager.cc:1939] T fd9d90e1ceab443dbec12b2c81b975e8 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:57.891753 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 219) took 5691 ms. Trace:
I20260810 23:32:57.881251 15091 raft_consensus.cc:482] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:57.905707 14121 rpcz_store.cc:276] 0810 23:32:52.200912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.204159 (+ 3247us) service_pool.cc:227] Handling call
0810 23:32:57.385977 (+5181818us) ts_tablet_manager.cc:1145] Deleting tablet fd9d90e1ceab443dbec12b2c81b975e8
0810 23:32:57.386028 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:57.704907 (+318879us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:57.887356 (+182449us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:57.891682 (+ 4326us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5154187}
I20260810 23:32:57.911387 15091 raft_consensus.cc:504] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:57.912894 14121 tablet_replica.cc:330] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:57.914670 15091 leader_election.cc:288] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:57.917057 14121 raft_consensus.cc:2229] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:57.903288 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea with delete_type TABLET_DATA_DELETED (replaced by tablet 7a0e69195596486fb03256baa22446e0) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:57.920990 14121 raft_consensus.cc:2258] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:57.927838 14121 ts_tablet_manager.cc:1905] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:57.978312 14121 ts_tablet_manager.cc:1918] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:57.982475 13514 tablet_bootstrap.cc:492] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:57.991132 15082 ts_tablet_manager.cc:1918] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:57.991655 14121 log.cc:1196] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/423e8d583d704fa9834026ebcd57e17e
I20260810 23:32:57.997229 14121 ts_tablet_manager.cc:1939] T 423e8d583d704fa9834026ebcd57e17e P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
W20260810 23:32:57.778586 12226 rpcz_store.cc:269] 0810 23:32:53.980697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:54.124617 (+143920us) service_pool.cc:227] Handling call
0810 23:32:54.164262 (+ 39645us) spinlock_profiling.cc:243] Waited 35.2 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:57.778073 (+3613811us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:57.253462 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:57.253945 (+ 483us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:57.253980 (+ 35us) write_op.cc:435] Acquired schema lock
0810 23:32:57.253992 (+ 12us) tablet.cc:660] Decoding operations
0810 23:32:57.254334 (+ 342us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:57.254386 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:57.254412 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:57.254598 (+ 186us) tablet.cc:699] Row locks acquired
0810 23:32:57.254610 (+ 12us) write_op.cc:260] PREPARE: finished
0810 23:32:57.254753 (+ 143us) write_op.cc:270] Start()
0810 23:32:57.254858 (+ 105us) write_op.cc:276] Timestamp: P: 1786404777254689 usec, L: 0
0810 23:32:57.254873 (+ 15us) op_driver.cc:331] REPLICATION: starting
0810 23:32:57.255540 (+ 667us) log.cc:842] Serialized 356 byte log entry
0810 23:32:57.272349 (+ 16809us) op_driver.cc:444] REPLICATION: finished
0810 23:32:57.272769 (+ 420us) write_op.cc:301] APPLY: starting
0810 23:32:57.272852 (+ 83us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:57.272993 (+ 141us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:57.273104 (+ 111us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:57.273377 (+ 273us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:57.273553 (+ 176us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:57.273592 (+ 39us) write_op.cc:312] APPLY: finished
0810 23:32:57.273807 (+ 215us) log.cc:842] Serialized 23 byte log entry
0810 23:32:57.273964 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:57.274044 (+ 80us) write_op.cc:454] Released schema lock
0810 23:32:57.274078 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:57.274269 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:57.274294 (+ 25us) write_op.cc:454] Released schema lock
Related trace 'op':
0810 23:32:57.314433 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:57.314935 (+ 502us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:57.314972 (+ 37us) write_op.cc:435] Acquired schema lock
0810 23:32:57.314988 (+ 16us) tablet.cc:660] Decoding operations
0810 23:32:57.315319 (+ 331us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:57.315368 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:57.315393 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:57.315562 (+ 169us) tablet.cc:699] Row locks acquired
0810 23:32:57.315580 (+ 18us) write_op.cc:260] PREPARE: finished
0810 23:32:57.315723 (+ 143us) write_op.cc:270] Start()
0810 23:32:57.316039 (+ 316us) write_op.cc:276] Timestamp: P: 1786404777315654 usec, L: 0
0810 23:32:57.316062 (+ 23us) op_driver.cc:331] REPLICATION: starting
0810 23:32:57.317115 (+ 1053us) log.cc:842] Serialized 356 byte log entry
0810 23:32:57.327290 (+ 10175us) op_driver.cc:444] REPLICATION: finished
0810 23:32:57.340230 (+ 12940us) write_op.cc:301] APPLY: starting
0810 23:32:57.340304 (+ 74us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:57.340423 (+ 119us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:57.340524 (+ 101us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:57.340755 (+ 231us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:57.340956 (+ 201us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:57.341024 (+ 68us) write_op.cc:312] APPLY: finished
0810 23:32:57.341315 (+ 291us) log.cc:842] Serialized 23 byte log entry
0810 23:32:57.341527 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:57.341624 (+ 97us) write_op.cc:454] Released schema lock
0810 23:32:57.341679 (+ 55us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:57.341933 (+ 254us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:57.341979 (+ 46us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":154974,"reported_tablets":12,"spinlock_wait_cycles":80942336,"child_traces":[["op",{"apply.queue_time_us":222,"apply.run_cpu_time_us":1713,"apply.run_wall_time_us":1712,"num_ops":1,"prepare.queue_time_us":6127,"prepare.run_cpu_time_us":2492,"prepare.run_wall_time_us":3796,"raft.queue_time_us":426,"raft.run_cpu_time_us":764,"raft.run_wall_time_us":765,"replication_time_us":17382}],["op",{"apply.queue_time_us":12713,"apply.run_cpu_time_us":1985,"apply.run_wall_time_us":1985,"num_ops":1,"prepare.queue_time_us":131,"prepare.run_cpu_time_us":2613,"prepare.run_wall_time_us":5804,"raft.queue_time_us":111,"raft.run_cpu_time_us":859,"raft.run_wall_time_us":860,"replication_time_us":11140}]]}
W20260810 23:32:57.877532 12228 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 39) took 2580 ms (client timeout 1000 ms). Trace:
I20260810 23:32:57.911396 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet fd9d90e1ceab443dbec12b2c81b975e8 (table default.Table_dc5d83ea7f1441fa9e2b2536bab49724 [id=aef6fd3aebac448e96d5942fd12836ca]) successfully deleted
W20260810 23:32:58.001001 12215 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 89651.2us
I20260810 23:32:57.950642 12223 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 320) took 2221 ms. Trace:
I20260810 23:32:57.954969 12235 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 319) took 2299 ms. Trace:
I20260810 23:32:58.007203 13691 tablet_bootstrap.cc:492] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:58.021981 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bce222cd5ab41fdb12fff3fe3dfbd3c" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:58.026988 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_d9fd2a26e54c483ebbc082b5a4ce014f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:58.029654 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9fd2a26e54c483ebbc082b5a4ce014f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:58.035615 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684caf5aaa8347e8b4e02323318fb0b7" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:58.036664 12240 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_3c4e5ed8ffd1426b81a1208fe3f0ba0c" } new_table_name: "default.Table_d028b6b163de469a9325b241cb5701a1" modify_external_catalogs: true
I20260810 23:32:58.042802 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240adfa4be844518bbc9624d0e408789" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:58.007880 13691 ts_tablet_manager.cc:1397] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.845s user 0.012s sys 0.001s
I20260810 23:32:58.049587 13691 raft_consensus.cc:348] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:57.657361 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684caf5aaa8347e8b4e02323318fb0b7" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:57.661321 13273 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c45b71432a45269cd6a106fc5f7292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:57.663347 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0dd19112504f8ca57f64dcf0bd50a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:57.665546 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d4c879ca6a402dbafe91940e7b367a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:39 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:57.750301 15093 raft_consensus.cc:482] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:58.066016 15093 raft_consensus.cc:504] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.067463 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea with delete_type TABLET_DATA_DELETED (replaced by tablet 7a0e69195596486fb03256baa22446e0) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:58.068311 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ad5d63a2104995990a86a000963cdc with delete_type TABLET_DATA_DELETED (replaced by tablet 8fb2ee1c9b41478da001aa67d7a7ad3c) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:58.068681 15093 leader_election.cc:288] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
W20260810 23:32:58.008932 12228 rpcz_store.cc:269] 0810 23:32:55.297907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:55.453093 (+155186us) service_pool.cc:227] Handling call
0810 23:32:57.877403 (+2424310us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:57.135509 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:57.136007 (+ 498us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:57.136067 (+ 60us) write_op.cc:435] Acquired schema lock
0810 23:32:57.136086 (+ 19us) tablet.cc:660] Decoding operations
0810 23:32:57.136455 (+ 369us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:57.136512 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:57.136540 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:57.136740 (+ 200us) tablet.cc:699] Row locks acquired
0810 23:32:57.136756 (+ 16us) write_op.cc:260] PREPARE: finished
0810 23:32:57.136908 (+ 152us) write_op.cc:270] Start()
0810 23:32:57.137015 (+ 107us) write_op.cc:276] Timestamp: P: 1786404777136849 usec, L: 0
0810 23:32:57.137032 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:32:57.137686 (+ 654us) log.cc:842] Serialized 536 byte log entry
0810 23:32:57.148459 (+ 10773us) op_driver.cc:444] REPLICATION: finished
0810 23:32:57.148866 (+ 407us) write_op.cc:301] APPLY: starting
0810 23:32:57.148931 (+ 65us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:57.149035 (+ 104us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:57.149134 (+ 99us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:57.149366 (+ 232us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:57.149511 (+ 145us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:57.149557 (+ 46us) write_op.cc:312] APPLY: finished
0810 23:32:57.149778 (+ 221us) log.cc:842] Serialized 23 byte log entry
0810 23:32:57.149926 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:57.150000 (+ 74us) write_op.cc:454] Released schema lock
0810 23:32:57.150033 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:57.150251 (+ 218us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:57.150273 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":232,"apply.run_cpu_time_us":1577,"apply.run_wall_time_us":1576,"num_ops":1,"prepare.queue_time_us":530,"prepare.run_cpu_time_us":2537,"prepare.run_wall_time_us":2536,"raft.queue_time_us":1500,"raft.run_cpu_time_us":800,"raft.run_wall_time_us":801,"replication_time_us":11320}]]}
I20260810 23:32:58.080323 15082 log.cc:1196] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/92b81e7056a441d2854ba5ab6b4c5433
I20260810 23:32:58.081590 15082 ts_tablet_manager.cc:1939] T 92b81e7056a441d2854ba5ab6b4c5433 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:58.003690 12223 rpcz_store.cc:276] 0810 23:32:55.729356 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.609720 (+1880364us) service_pool.cc:227] Handling call
0810 23:32:57.829811 (+220091us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:57.855263 (+ 25452us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:57.949699 (+ 94436us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:57.950074 (+ 375us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:57.950514 (+ 440us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:57.913919 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:57.914763 (+ 844us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:57.914795 (+ 32us) write_op.cc:435] Acquired schema lock
0810 23:32:57.914809 (+ 14us) tablet.cc:660] Decoding operations
0810 23:32:57.915454 (+ 645us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:57.915507 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:57.915542 (+ 35us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:57.915930 (+ 388us) tablet.cc:699] Row locks acquired
0810 23:32:57.915943 (+ 13us) write_op.cc:260] PREPARE: finished
0810 23:32:57.916274 (+ 331us) write_op.cc:270] Start()
0810 23:32:57.916425 (+ 151us) write_op.cc:276] Timestamp: P: 1786404777916197 usec, L: 0
0810 23:32:57.916444 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 23:32:57.917450 (+ 1006us) log.cc:842] Serialized 433 byte log entry
0810 23:32:57.931634 (+ 14184us) op_driver.cc:444] REPLICATION: finished
0810 23:32:57.932249 (+ 615us) write_op.cc:301] APPLY: starting
0810 23:32:57.932374 (+ 125us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:57.932630 (+ 256us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:57.932869 (+ 239us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:57.934370 (+ 1501us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:57.934605 (+ 235us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:57.934710 (+ 105us) write_op.cc:312] APPLY: finished
0810 23:32:57.935094 (+ 384us) log.cc:842] Serialized 29 byte log entry
0810 23:32:57.936148 (+ 1054us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:57.936281 (+ 133us) write_op.cc:454] Released schema lock
0810 23:32:57.936357 (+ 76us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:57.936699 (+ 342us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:57.936737 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":379,"apply.run_cpu_time_us":4047,"apply.run_wall_time_us":4758,"mutex_wait_us":56902,"num_ops":2,"prepare.queue_time_us":57102,"prepare.run_cpu_time_us":4024,"prepare.run_wall_time_us":43261,"raft.queue_time_us":155,"raft.run_cpu_time_us":1147,"raft.run_wall_time_us":2219,"replication_time_us":15069}]]}
I20260810 23:32:58.085074 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3049bcdc6e43c795961c74f0d008f5" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:58.085706 12806 raft_consensus.cc:2454] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:58.005523 12235 rpcz_store.cc:276] 0810 23:32:55.655322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.579861 (+1924539us) service_pool.cc:227] Handling call
0810 23:32:57.829426 (+249565us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:57.843382 (+ 13956us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:57.953643 (+110261us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:57.953676 (+ 33us) write_op.cc:454] Released schema lock
0810 23:32:57.953977 (+ 301us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:57.954508 (+ 531us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:57.954859 (+ 351us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:57.878847 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:57.879643 (+ 796us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:57.879700 (+ 57us) write_op.cc:435] Acquired schema lock
0810 23:32:57.879752 (+ 52us) tablet.cc:660] Decoding operations
0810 23:32:57.880527 (+ 775us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:57.880603 (+ 76us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:57.880679 (+ 76us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:57.881111 (+ 432us) tablet.cc:699] Row locks acquired
0810 23:32:57.881158 (+ 47us) write_op.cc:260] PREPARE: finished
0810 23:32:57.881451 (+ 293us) write_op.cc:270] Start()
0810 23:32:57.881661 (+ 210us) write_op.cc:276] Timestamp: P: 1786404777881390 usec, L: 0
0810 23:32:57.881713 (+ 52us) op_driver.cc:331] REPLICATION: starting
0810 23:32:57.882578 (+ 865us) log.cc:842] Serialized 433 byte log entry
0810 23:32:57.897045 (+ 14467us) op_driver.cc:444] REPLICATION: finished
0810 23:32:57.897533 (+ 488us) write_op.cc:301] APPLY: starting
0810 23:32:57.897687 (+ 154us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:57.897935 (+ 248us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:57.898190 (+ 255us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:57.899268 (+ 1078us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:57.899613 (+ 345us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:57.899755 (+ 142us) write_op.cc:312] APPLY: finished
0810 23:32:57.902395 (+ 2640us) log.cc:842] Serialized 29 byte log entry
0810 23:32:57.902735 (+ 340us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:57.902893 (+ 158us) write_op.cc:454] Released schema lock
0810 23:32:57.902986 (+ 93us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":184,"apply.run_cpu_time_us":3684,"apply.run_wall_time_us":5784,"num_ops":2,"prepare.queue_time_us":33774,"prepare.run_cpu_time_us":4278,"prepare.run_wall_time_us":32220,"raft.queue_time_us":198,"raft.run_cpu_time_us":1114,"raft.run_wall_time_us":1114,"replication_time_us":15087}]]}
I20260810 23:32:58.087474 12235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_3e3ca6011776476baa1807ba610d0d3d" } modify_external_catalogs: true
I20260810 23:32:58.054982 12237 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 321) took 2308 ms. Trace:
I20260810 23:32:58.088135 12237 rpcz_store.cc:276] 0810 23:32:55.746816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.774365 (+2027549us) service_pool.cc:227] Handling call
0810 23:32:57.957306 (+182941us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:57.959394 (+ 2088us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:58.054120 (+ 94726us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:58.054523 (+ 403us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:58.054880 (+ 357us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:57.968392 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:57.969249 (+ 857us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:57.969330 (+ 81us) write_op.cc:435] Acquired schema lock
0810 23:32:57.969344 (+ 14us) tablet.cc:660] Decoding operations
0810 23:32:57.969786 (+ 442us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:57.969882 (+ 96us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:57.969936 (+ 54us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:57.970349 (+ 413us) tablet.cc:699] Row locks acquired
0810 23:32:57.970364 (+ 15us) write_op.cc:260] PREPARE: finished
0810 23:32:57.970673 (+ 309us) write_op.cc:270] Start()
0810 23:32:57.970864 (+ 191us) write_op.cc:276] Timestamp: P: 1786404777970596 usec, L: 0
0810 23:32:57.970881 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:32:57.971961 (+ 1080us) log.cc:842] Serialized 433 byte log entry
0810 23:32:58.038096 (+ 66135us) op_driver.cc:444] REPLICATION: finished
0810 23:32:58.038880 (+ 784us) write_op.cc:301] APPLY: starting
0810 23:32:58.039008 (+ 128us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:58.039187 (+ 179us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:58.039404 (+ 217us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:58.040776 (+ 1372us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:58.041504 (+ 728us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:58.041562 (+ 58us) write_op.cc:312] APPLY: finished
0810 23:32:58.041918 (+ 356us) log.cc:842] Serialized 29 byte log entry
0810 23:32:58.043248 (+ 1330us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:58.043392 (+ 144us) write_op.cc:454] Released schema lock
0810 23:32:58.043450 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:58.043697 (+ 247us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:58.043720 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":519,"apply.run_cpu_time_us":3447,"apply.run_wall_time_us":5051,"num_ops":2,"prepare.queue_time_us":6460,"prepare.run_cpu_time_us":4294,"prepare.run_wall_time_us":90615,"raft.queue_time_us":142,"raft.run_cpu_time_us":1334,"raft.run_wall_time_us":2657,"replication_time_us":66998}]]}
I20260810 23:32:58.092867 12228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_620c572fc75b44fe8ca892789875c1f7" } modify_external_catalogs: true
I20260810 23:32:58.094318 12228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_620c572fc75b44fe8ca892789875c1f7" } modify_external_catalogs: true
I20260810 23:32:58.093725 12235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_3e3ca6011776476baa1807ba610d0d3d" } modify_external_catalogs: true
I20260810 23:32:58.076882 13514 tablet_bootstrap.cc:654] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:58.097770 12226 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 322) took 2339 ms. Trace:
I20260810 23:32:58.098126 12226 rpcz_store.cc:276] 0810 23:32:55.758548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.999845 (+2241297us) service_pool.cc:227] Handling call
0810 23:32:58.026752 (+ 26907us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7156 00007f0ce6e24034 00007f0ce6e23fc2 00007f0ce6e23ef1 00007f0ce6e23e6c 00007f0ce6e23122 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:58.030341 (+ 3589us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:58.032808 (+ 2467us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:58.096996 (+ 64188us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:58.097361 (+ 365us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:58.097668 (+ 307us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:58.059261 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:58.059790 (+ 529us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:58.059820 (+ 30us) write_op.cc:435] Acquired schema lock
0810 23:32:58.059850 (+ 30us) tablet.cc:660] Decoding operations
0810 23:32:58.060383 (+ 533us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:58.060431 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:58.060457 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:58.060714 (+ 257us) tablet.cc:699] Row locks acquired
0810 23:32:58.060733 (+ 19us) write_op.cc:260] PREPARE: finished
0810 23:32:58.060896 (+ 163us) write_op.cc:270] Start()
0810 23:32:58.061009 (+ 113us) write_op.cc:276] Timestamp: P: 1786404778060836 usec, L: 0
0810 23:32:58.061029 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:32:58.061827 (+ 798us) log.cc:842] Serialized 433 byte log entry
0810 23:32:58.093384 (+ 31557us) op_driver.cc:444] REPLICATION: finished
0810 23:32:58.093985 (+ 601us) write_op.cc:301] APPLY: starting
0810 23:32:58.094053 (+ 68us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:58.094201 (+ 148us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:58.094369 (+ 168us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:58.095118 (+ 749us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:58.095317 (+ 199us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:58.095385 (+ 68us) write_op.cc:312] APPLY: finished
0810 23:32:58.095683 (+ 298us) log.cc:842] Serialized 29 byte log entry
0810 23:32:58.096492 (+ 809us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:58.096585 (+ 93us) write_op.cc:454] Released schema lock
0810 23:32:58.096628 (+ 43us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":58988800,"child_traces":[["op",{"apply.queue_time_us":375,"apply.run_cpu_time_us":2428,"apply.run_wall_time_us":6503,"num_ops":2,"prepare.queue_time_us":24610,"raft.queue_time_us":19427,"raft.run_cpu_time_us":861,"raft.run_wall_time_us":1855,"replication_time_us":32241}]]}
I20260810 23:32:58.127533 13514 tablet_bootstrap.cc:492] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:58.128085 13514 ts_tablet_manager.cc:1397] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.146s user 0.011s sys 0.019s
I20260810 23:32:58.137429 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 92b81e7056a441d2854ba5ab6b4c5433 (table default.Table_e0faeb2fda9f4af48212613473c88681 [id=d8d3c66123cd4516a59a0ab8ae4d47b9]) successfully deleted
W20260810 23:32:58.140092 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet fd9d90e1ceab443dbec12b2c81b975e8 because the tablet was not found. No further retry: Not found: Tablet not found: fd9d90e1ceab443dbec12b2c81b975e8
W20260810 23:32:58.142669 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet ca0dd19112504f8ca57f64dcf0bd50a8 because the tablet was not found. No further retry: Not found: Tablet not found: ca0dd19112504f8ca57f64dcf0bd50a8
I20260810 23:32:58.144308 15082 tablet_replica.cc:330] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
W20260810 23:32:58.148586 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 391940us
W20260810 23:32:58.154556 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 42517f11422b42c39db282e2c5f584a3 because the tablet was not found. No further retry: Not found: Tablet not found: 42517f11422b42c39db282e2c5f584a3
I20260810 23:32:58.153836 13974 tablet_replica.cc:330] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:58.158351 13974 raft_consensus.cc:2229] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:58.159109 13974 raft_consensus.cc:2258] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:58.163746 12237 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_fc0d875790454fcba23aa9c7e2ee1c4e" } new_table_name: "default.Table_9edef8525efa4d13af777efceb37a603" modify_external_catalogs: true
I20260810 23:32:58.165889 15082 raft_consensus.cc:2229] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:58.148000 15093 raft_consensus.cc:2781] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:58.166501 15082 raft_consensus.cc:2258] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:58.171979 15093 raft_consensus.cc:686] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:58.173175 15093 consensus_queue.cc:237] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.200883 13974 ts_tablet_manager.cc:1905] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:58.055083 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 221) took 5826 ms. Trace:
I20260810 23:32:58.081929 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb2ee1c9b41478da001aa67d7a7ad3c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:58.228266 14121 rpcz_store.cc:276] 0810 23:32:52.229208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.229353 (+ 145us) service_pool.cc:227] Handling call
0810 23:32:57.912661 (+5683308us) ts_tablet_manager.cc:1145] Deleting tablet 423e8d583d704fa9834026ebcd57e17e
0810 23:32:57.912725 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:57.954432 (+ 41707us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:57.978011 (+ 23579us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:58.055003 (+ 76992us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5679095}
I20260810 23:32:58.229689 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 222) took 5997 ms. Trace:
I20260810 23:32:58.228974 15087 raft_consensus.cc:504] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.230062 14121 rpcz_store.cc:276] 0810 23:32:52.232386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.232542 (+ 156us) service_pool.cc:227] Handling call
0810 23:32:58.229370 (+5996828us) ts_tablet_manager.cc:1145] Deleting tablet 874ca0c3586b41a8a3074d8988813633
0810 23:32:58.229434 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:58.229629 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5994766}
I20260810 23:32:58.231215 14121 tablet_replica.cc:330] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:32:58.232607 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 423e8d583d704fa9834026ebcd57e17e (table default.Table_514d038dba3a4fc399b97e7edc254ad9 [id=53909ddaaede4deb97cec7ad02020c5d]) successfully deleted
I20260810 23:32:58.233453 14121 raft_consensus.cc:2229] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
W20260810 23:32:58.233999 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 874ca0c3586b41a8a3074d8988813633 because the tablet was not found. No further retry: Not found: Tablet not found: 874ca0c3586b41a8a3074d8988813633
I20260810 23:32:58.234505 14121 raft_consensus.cc:2258] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:58.237857 14121 ts_tablet_manager.cc:1905] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:40.608541 14113 debug-util.cc:398] Leaking SignalData structure 0x7b0800014860 after lost signal to thread 13549
I20260810 23:32:58.228235 13691 raft_consensus.cc:374] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 23:32:58.243789 14150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:58.244127 13691 raft_consensus.cc:729] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
W20260810 23:32:58.230737 13549 thread.cc:636] OpenStack (cloud detector) Time spent creating pthread: real 18.972s user 1.079s sys 1.985s
W20260810 23:32:58.244899 13549 thread.cc:602] OpenStack (cloud detector) Time spent starting thread: real 18.987s user 1.079s sys 1.985s
I20260810 23:32:58.245446 13691 consensus_queue.cc:260] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:32:39.143086 14126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:58.260537 13549 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260810 23:32:58.262096 12698 leader_election.cc:341] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:58.276217 14138 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 23:32:58.295838 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_a3a059d2601345d68eadf189772bcde5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:58.250222 13212 leader_election.cc:341] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:32:58.298027 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3a059d2601345d68eadf189772bcde5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:32:58.298875 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 48890.5us
I20260810 23:32:58.296108 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_afb094287f0e4e88aefdd128d2fd7ef8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:58.301321 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afb094287f0e4e88aefdd128d2fd7ef8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:58.304518 13549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:58.267382 15082 ts_tablet_manager.cc:1905] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:58.301342 12700 leader_election.cc:341] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:58.312525 12700 leader_election.cc:302] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:58.313668 14979 raft_consensus.cc:2726] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:32:58.318753 12700 leader_election.cc:341] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:58.319006 13514 raft_consensus.cc:348] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.319769 13514 raft_consensus.cc:374] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:58.320101 13514 raft_consensus.cc:729] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:58.280536 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240adfa4be844518bbc9624d0e408789" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:58.322463 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3049bcdc6e43c795961c74f0d008f5" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:58.279191 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb2ee1c9b41478da001aa67d7a7ad3c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:58.323158 13054 raft_consensus.cc:2454] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:58.324981 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bce222cd5ab41fdb12fff3fe3dfbd3c" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:32:58.321751 13514 consensus_queue.cc:260] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.328122 13549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:58.231972 15087 leader_election.cc:288] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:58.330905 13514 ts_tablet_manager.cc:1428] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.203s user 0.006s sys 0.036s
I20260810 23:32:58.332432 13665 tablet_bootstrap.cc:492] T cd24471dd52149d395808493ad9fa68c P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:58.333061 13665 ts_tablet_manager.cc:1397] T cd24471dd52149d395808493ad9fa68c P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.927s user 0.023s sys 0.004s
I20260810 23:32:58.344631 13549 hybrid_clock.cc:648] HybridClock initialized: now 1786404778344395 us; error 220 us; skew 500 ppm
I20260810 23:32:58.346654 13549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:58.346848 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3049bcdc6e43c795961c74f0d008f5" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:58.347795 13327 raft_consensus.cc:2454] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:58.355717 13691 ts_tablet_manager.cc:1428] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.310s user 0.009s sys 0.000s
I20260810 23:32:58.374379 15082 ts_tablet_manager.cc:1918] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:58.375226 15082 log.cc:1196] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/4cdec13ee70d402fa3a08119e7bf1313
I20260810 23:32:58.376572 15082 ts_tablet_manager.cc:1939] T 4cdec13ee70d402fa3a08119e7bf1313 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:58.385804 12952 leader_election.cc:302] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:58.386831 15014 raft_consensus.cc:2781] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:58.387285 15014 raft_consensus.cc:482] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:58.388708 15014 raft_consensus.cc:3037] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:58.389053 13974 ts_tablet_manager.cc:1918] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:58.389508 13974 log.cc:1196] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/9c29b11192604ac18c21b7bdc0238fb3
I20260810 23:32:58.389690 13665 raft_consensus.cc:348] T cd24471dd52149d395808493ad9fa68c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.390326 13974 ts_tablet_manager.cc:1939] T 9c29b11192604ac18c21b7bdc0238fb3 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:58.390516 13665 raft_consensus.cc:374] T cd24471dd52149d395808493ad9fa68c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:58.390816 13665 raft_consensus.cc:729] T cd24471dd52149d395808493ad9fa68c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:58.403012 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 257) took 1135 ms. Trace:
I20260810 23:32:58.404367 13665 consensus_queue.cc:260] T cd24471dd52149d395808493ad9fa68c P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.404765 13974 rpcz_store.cc:276] 0810 23:32:57.267867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.268571 (+ 704us) service_pool.cc:227] Handling call
0810 23:32:58.153614 (+885043us) ts_tablet_manager.cc:1145] Deleting tablet 9c29b11192604ac18c21b7bdc0238fb3
0810 23:32:58.153668 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:58.338456 (+184788us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:58.388747 (+ 50291us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:58.402924 (+ 14177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":883572}
I20260810 23:32:58.407583 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 255) took 1140 ms. Trace:
I20260810 23:32:58.407955 13974 rpcz_store.cc:276] 0810 23:32:57.267230 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.270149 (+ 2919us) service_pool.cc:227] Handling call
0810 23:32:58.407240 (+1137091us) ts_tablet_manager.cc:1145] Deleting tablet b261673faa964b65af3576c89e302864
0810 23:32:58.407297 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:58.407516 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136391}
I20260810 23:32:58.408368 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 9c29b11192604ac18c21b7bdc0238fb3 (table default.Table_dd2cd30e5e2349de9180be01866dc513 [id=8754da7be72b4eef80daacdcdfde82a7]) successfully deleted
I20260810 23:32:58.409449 13974 tablet_replica.cc:330] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
W20260810 23:32:58.409372 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet b261673faa964b65af3576c89e302864 because the tablet was not found. No further retry: Not found: Tablet not found: b261673faa964b65af3576c89e302864
I20260810 23:32:58.414989 13974 raft_consensus.cc:2229] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:58.415959 13974 raft_consensus.cc:2258] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:58.418790 13974 ts_tablet_manager.cc:1905] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:58.422175 12237 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff732fd231a74846b12686329ffe53c3 consensus state has no leader
I20260810 23:32:58.422307 13665 ts_tablet_manager.cc:1428] T cd24471dd52149d395808493ad9fa68c P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.089s user 0.005s sys 0.004s
I20260810 23:32:58.432112 15014 raft_consensus.cc:504] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.433084 13549 webserver.cc:460] Webserver started at http://127.11.226.189:37775/ using document root <none> and password file <none>
I20260810 23:32:58.436470 13665 tablet_bootstrap.cc:492] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:58.436792 13549 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:58.437502 13549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:58.437788 14979 raft_consensus.cc:482] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:58.438632 14979 raft_consensus.cc:504] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.442389 15130 raft_consensus.cc:482] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:58.443161 15130 raft_consensus.cc:504] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.446864 15130 leader_election.cc:288] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:58.448261 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3049bcdc6e43c795961c74f0d008f5" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:58.449091 14979 leader_election.cc:288] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:58.449183 13327 raft_consensus.cc:3037] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:58.474081 12785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3049bcdc6e43c795961c74f0d008f5" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:58.475270 12785 raft_consensus.cc:2454] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:58.477216 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd24471dd52149d395808493ad9fa68c" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:58.478329 13334 raft_consensus.cc:2454] T cd24471dd52149d395808493ad9fa68c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:58.481320 13665 tablet_bootstrap.cc:654] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:58.491061 13974 ts_tablet_manager.cc:1918] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:58.490929 12952 leader_election.cc:302] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:58.491715 13974 log.cc:1196] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/92b81e7056a441d2854ba5ab6b4c5433
I20260810 23:32:58.492704 13974 ts_tablet_manager.cc:1939] T 92b81e7056a441d2854ba5ab6b4c5433 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:58.492624 15133 raft_consensus.cc:2781] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:58.493211 15133 raft_consensus.cc:482] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:58.493892 15133 raft_consensus.cc:3037] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:58.498162 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 261) took 1133 ms. Trace:
I20260810 23:32:58.500512 13974 rpcz_store.cc:276] 0810 23:32:57.364343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.364534 (+ 191us) service_pool.cc:227] Handling call
0810 23:32:58.409231 (+1044697us) ts_tablet_manager.cc:1145] Deleting tablet 92b81e7056a441d2854ba5ab6b4c5433
0810 23:32:58.409293 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:58.470339 (+ 61046us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:58.490787 (+ 20448us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:58.497879 (+ 7092us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4304,"tablet-delete.queue_time_us":1043903}
I20260810 23:32:58.503543 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 92b81e7056a441d2854ba5ab6b4c5433 (table default.Table_e0faeb2fda9f4af48212613473c88681 [id=d8d3c66123cd4516a59a0ab8ae4d47b9]) successfully deleted
I20260810 23:32:58.504419 13691 tablet_bootstrap.cc:492] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:58.507910 13974 tablet_replica.cc:330] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:58.510588 13974 raft_consensus.cc:2229] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:58.511981 13974 raft_consensus.cc:2258] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:58.514976 13974 ts_tablet_manager.cc:1905] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:58.518769 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 4d697e7533af4fc0a15cb41f3fcce4db because the tablet was not found. No further retry: Not found: Tablet not found: 4d697e7533af4fc0a15cb41f3fcce4db
W20260810 23:32:58.519551 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 4d0d9227d8574de8bd73041f3e8043a0 because the tablet was not found. No further retry: Not found: Tablet not found: 4d0d9227d8574de8bd73041f3e8043a0
I20260810 23:32:58.521279 15133 raft_consensus.cc:504] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.487857 13327 raft_consensus.cc:2454] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:58.527375 12952 leader_election.cc:302] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:58.529224 15130 raft_consensus.cc:2781] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:58.529896 15130 raft_consensus.cc:686] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:58.531528 13549 fs_manager.cc:709] Time spent opening directory manager: real 0.074s user 0.014s sys 0.010s
I20260810 23:32:58.531175 15130 consensus_queue.cc:237] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.541467 12234 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:32:58.542448 12234 catalog_manager.cc:5495] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f26517150f1c480a877f15aec71d020b (127.11.226.130). New cstate: current_term: 1 leader_uuid: "f26517150f1c480a877f15aec71d020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } }
I20260810 23:32:58.543962 12234 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet c01ede4b52c3498a924340b39c3c2435 (table default.Table_ebe848b2f9ba4077a556369c87aa436a [id=ae7b5faec4c34d35981e0a1d464202ef]). Expected version 1 got 0
I20260810 23:32:58.545467 15133 leader_election.cc:288] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:58.545099 12234 catalog_manager.cc:5495] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } }
I20260810 23:32:58.543371 15014 leader_election.cc:288] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:58.547055 12234 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5]). Expected version 2 got 0
W20260810 23:32:58.553010 12700 leader_election.cc:341] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:58.558260 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd24471dd52149d395808493ad9fa68c" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:58.558971 12805 raft_consensus.cc:2454] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:32:58.551668 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd24471dd52149d395808493ad9fa68c" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:32:58.555022 13212 leader_election.cc:302] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:58.560351 13017 tablet_service.cc:1467] Processing CreateTablet for tablet e27986aa12e44ace92b335421a457348 (DEFAULT_TABLE table=default.Table_d3429684c4a7448caafcc4052426b361 [id=9e7f38ba26c14a66bc36404e70351094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:58.561980 13974 ts_tablet_manager.cc:1918] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:58.562466 13974 log.cc:1196] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/423e8d583d704fa9834026ebcd57e17e
I20260810 23:32:58.562542 15180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:58.563128 13335 raft_consensus.cc:3037] T cd24471dd52149d395808493ad9fa68c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:58.563426 13974 ts_tablet_manager.cc:1939] T 423e8d583d704fa9834026ebcd57e17e P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
W20260810 23:32:58.564850 13212 leader_election.cc:341] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:58.568913 13549 fs_manager.cc:725] Time spent opening block manager: real 0.032s user 0.003s sys 0.009s
I20260810 23:32:58.568310 15153 consensus_queue.cc:1035] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:58.569577 13514 tablet_bootstrap.cc:492] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:58.572147 15156 raft_consensus.cc:2741] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260810 23:32:58.572957 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 268) took 1025 ms. Trace:
I20260810 23:32:58.573553 15082 rpcz_store.cc:276] 0810 23:32:57.547103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.549695 (+ 2592us) service_pool.cc:227] Handling call
0810 23:32:58.143850 (+594155us) ts_tablet_manager.cc:1145] Deleting tablet 4cdec13ee70d402fa3a08119e7bf1313
0810 23:32:58.143950 (+ 100us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:58.358314 (+214364us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:58.373947 (+ 15633us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:58.572802 (+198855us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":593154}
I20260810 23:32:58.576269 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 4cdec13ee70d402fa3a08119e7bf1313 (table default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]) successfully deleted
I20260810 23:32:58.577940 15082 tablet_replica.cc:330] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:58.583444 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bce222cd5ab41fdb12fff3fe3dfbd3c" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:58.569490 13549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/wal
uuid: "198f534261994047a68e1d509bb2fd8b"
format_stamp: "Formatted at 2026-08-10 23:32:07 on dist-test-slave-kvfs"
I20260810 23:32:58.584268 12806 raft_consensus.cc:3037] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:58.584210 13549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:58.590775 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27986aa12e44ace92b335421a457348. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:58.595207 12698 leader_election.cc:341] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:58.595799 12698 leader_election.cc:302] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:58.597338 14979 raft_consensus.cc:2726] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:58.577956 13274 tablet_service.cc:1467] Processing CreateTablet for tablet e27986aa12e44ace92b335421a457348 (DEFAULT_TABLE table=default.Table_d3429684c4a7448caafcc4052426b361 [id=9e7f38ba26c14a66bc36404e70351094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:58.600003 13274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27986aa12e44ace92b335421a457348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:58.605239 13665 tablet_bootstrap.cc:492] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:58.613914 12237 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_d54c3da29f874f2e83e5a10727950256"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:58.617604 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 423e8d583d704fa9834026ebcd57e17e (table default.Table_514d038dba3a4fc399b97e7edc254ad9 [id=53909ddaaede4deb97cec7ad02020c5d]) successfully deleted
W20260810 23:32:58.618489 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet fd9d90e1ceab443dbec12b2c81b975e8 because the tablet was not found. No further retry: Not found: Tablet not found: fd9d90e1ceab443dbec12b2c81b975e8
W20260810 23:32:58.619277 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 08c45b71432a45269cd6a106fc5f7292 because the tablet was not found. No further retry: Not found: Tablet not found: 08c45b71432a45269cd6a106fc5f7292
W20260810 23:32:58.621073 12237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d54c3da29f874f2e83e5a10727950256 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:58.625880 12806 raft_consensus.cc:2454] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:32:58.634729 15130 consensus_queue.cc:1035] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:58.634971 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62705f99687340aaa1aca0942b28487f" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:58.639467 15082 raft_consensus.cc:2229] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:58.640290 15082 raft_consensus.cc:2258] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:32:58.640827 12698 leader_election.cc:341] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:58.641428 12698 leader_election.cc:302] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:32:58.642617 14979 raft_consensus.cc:2726] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:58.643748 15082 ts_tablet_manager.cc:1905] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:58.673539 12764 tablet_service.cc:1467] Processing CreateTablet for tablet e27986aa12e44ace92b335421a457348 (DEFAULT_TABLE table=default.Table_d3429684c4a7448caafcc4052426b361 [id=9e7f38ba26c14a66bc36404e70351094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:58.676949 13008 tablet_service.cc:1467] Processing CreateTablet for tablet 296330391f7e4af99675bc6d7fded8b8 (DEFAULT_TABLE table=default.Table_f07540d9f70a42a99acf3b3613c98d18 [id=ddf98dc81008484db03e1d98df5634c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:58.678535 13008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296330391f7e4af99675bc6d7fded8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:58.705616 15156 consensus_queue.cc:1035] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:58.608640 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62705f99687340aaa1aca0942b28487f" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:58.706511 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 45908.3us
I20260810 23:32:58.613672 13974 tablet_replica.cc:330] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:58.638463 13335 raft_consensus.cc:2454] T cd24471dd52149d395808493ad9fa68c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:32:58.677039 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 296330391f7e4af99675bc6d7fded8b8 (DEFAULT_TABLE table=default.Table_f07540d9f70a42a99acf3b3613c98d18 [id=ddf98dc81008484db03e1d98df5634c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:58.711131 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296330391f7e4af99675bc6d7fded8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:58.712774 13665 ts_tablet_manager.cc:1397] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.277s user 0.006s sys 0.016s
I20260810 23:32:58.721226 12952 leader_election.cc:302] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:58.723598 13974 raft_consensus.cc:2229] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:58.723416 15130 raft_consensus.cc:2781] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:58.724274 13974 raft_consensus.cc:2258] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:58.725275 15130 raft_consensus.cc:686] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:32:58.726729 15130 consensus_queue.cc:237] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.733403 13974 ts_tablet_manager.cc:1905] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:58.705662 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 43809.6us
W20260810 23:32:58.742514 12952 consensus_peers.cc:487] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:58.744463 13213 leader_election.cc:302] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:32:58.748160 15187 raft_consensus.cc:482] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:58.749369 15187 raft_consensus.cc:504] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.753201 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27986aa12e44ace92b335421a457348. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:58.788003 12958 consensus_peers.cc:487] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:58.796845 15156 raft_consensus.cc:2781] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
W20260810 23:32:58.796932 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 52955.8us
I20260810 23:32:58.797770 15156 raft_consensus.cc:686] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:58.800173 15156 consensus_queue.cc:237] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.813480 13974 ts_tablet_manager.cc:1918] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:58.822343 13974 log.cc:1196] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/ca0dd19112504f8ca57f64dcf0bd50a8
I20260810 23:32:58.823939 13974 ts_tablet_manager.cc:1939] T ca0dd19112504f8ca57f64dcf0bd50a8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:58.826422 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 296330391f7e4af99675bc6d7fded8b8 (DEFAULT_TABLE table=default.Table_f07540d9f70a42a99acf3b3613c98d18 [id=ddf98dc81008484db03e1d98df5634c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:58.829176 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296330391f7e4af99675bc6d7fded8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:58.842633 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 268) took 1182 ms. Trace:
I20260810 23:32:58.843858 13974 rpcz_store.cc:276] 0810 23:32:57.662830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.663039 (+ 209us) service_pool.cc:227] Handling call
0810 23:32:58.613501 (+950462us) ts_tablet_manager.cc:1145] Deleting tablet ca0dd19112504f8ca57f64dcf0bd50a8
0810 23:32:58.613535 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:58.777124 (+163589us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:58.813117 (+ 35993us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:58.842531 (+ 29414us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13410,"tablet-delete.queue_time_us":547903}
I20260810 23:32:58.845842 13974 tablet_replica.cc:330] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:58.846131 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet ca0dd19112504f8ca57f64dcf0bd50a8 (table default.Table_1e5f07908d444cf9af2cae2ea83ae7ae [id=a591a34da3514b87aa852684669a399f]) successfully deleted
I20260810 23:32:58.846889 13974 raft_consensus.cc:2229] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:58.847524 13974 raft_consensus.cc:2258] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:58.849983 13974 ts_tablet_manager.cc:1905] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:58.861858 12763 tablet_service.cc:1467] Processing CreateTablet for tablet 09bdc7c2aace45da9fce81c4904db5d4 (DEFAULT_TABLE table=default.Table_2e4541bbe3aa45c1a1c85533c2ed79f3 [id=c9327e173e574f75b4886311e0933acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:58.863183 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09bdc7c2aace45da9fce81c4904db5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:58.850445 13665 raft_consensus.cc:348] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.866503 13665 raft_consensus.cc:374] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:58.866801 13665 raft_consensus.cc:729] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:58.871454 12744 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:47536 (request call id 78) took 1082 ms. Trace:
I20260810 23:32:58.879405 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 09bdc7c2aace45da9fce81c4904db5d4 (DEFAULT_TABLE table=default.Table_2e4541bbe3aa45c1a1c85533c2ed79f3 [id=c9327e173e574f75b4886311e0933acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:58.880990 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09bdc7c2aace45da9fce81c4904db5d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:32:58.885336 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 97446.7us
I20260810 23:32:58.885552 12224 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:58.896732 13665 consensus_queue.cc:260] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:58.899533 13665 ts_tablet_manager.cc:1428] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.186s user 0.005s sys 0.001s
I20260810 23:32:58.906464 13974 ts_tablet_manager.cc:1918] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:58.906968 13974 log.cc:1196] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/c3d4c879ca6a402dbafe91940e7b367a
I20260810 23:32:58.907913 13974 ts_tablet_manager.cc:1939] T c3d4c879ca6a402dbafe91940e7b367a P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:58.910151 13514 tablet_bootstrap.cc:654] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:58.917840 13665 tablet_bootstrap.cc:492] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:58.921633 13974 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 269) took 1257 ms. Trace:
I20260810 23:32:58.922227 13974 rpcz_store.cc:276] 0810 23:32:57.664065 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.664406 (+ 341us) service_pool.cc:227] Handling call
0810 23:32:58.845629 (+1181223us) ts_tablet_manager.cc:1145] Deleting tablet c3d4c879ca6a402dbafe91940e7b367a
0810 23:32:58.845685 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:58.872571 (+ 26886us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:58.906156 (+ 33585us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:58.921519 (+ 15363us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":779809}
I20260810 23:32:58.923843 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet c3d4c879ca6a402dbafe91940e7b367a (table default.Table_6e13e5c0a9da4cc5a1b85b4a247f8a3f [id=820d73489eed49d8a72357b3ac31300d]) successfully deleted
I20260810 23:32:58.880652 13691 tablet_bootstrap.cc:654] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:58.932639 13665 tablet_bootstrap.cc:654] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
W20260810 23:32:58.936580 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea already in progress: creating tablet
W20260810 23:32:58.940436 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 13ad5d63a2104995990a86a000963cdc because the tablet was not found. No further retry: Not found: Tablet not found: 13ad5d63a2104995990a86a000963cdc
W20260810 23:32:58.915026 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 111943us
I20260810 23:32:58.958868 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3049bcdc6e43c795961c74f0d008f5" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:58.959568 12806 raft_consensus.cc:3037] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:58.963119 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet 240adfa4be844518bbc9624d0e408789 with delete_type TABLET_DATA_DELETED (replaced by tablet 4344987a6c90428fbdf50765a4631455) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:58.965478 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd24471dd52149d395808493ad9fa68c" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:58.965701 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdec13ee70d402fa3a08119e7bf1313 with delete_type TABLET_DATA_DELETED (replaced by tablet d603569c54ef4ffba9941d0cc3e0f97f) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:58.966331 12805 raft_consensus.cc:3037] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:58.969103 12218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.188:45477, user_credentials={real_user=slave}} blocked reactor thread for 85516.6us
I20260810 23:32:58.977396 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet 62705f99687340aaa1aca0942b28487f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:48 UTC) from {username='slave'} at 127.0.0.1:47546
W20260810 23:32:58.978875 12234 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 63) took 1199 ms (client timeout 1000 ms). Trace:
I20260810 23:32:58.945075 15187 leader_election.cc:288] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:58.988451 12224 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet c01ede4b52c3498a924340b39c3c2435 (table default.Table_ebe848b2f9ba4077a556369c87aa436a [id=ae7b5faec4c34d35981e0a1d464202ef]). Expected version 1 got 0
I20260810 23:32:58.991688 12805 raft_consensus.cc:2454] T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
W20260810 23:32:58.992604 12214 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 45e39a7b58ac402da7c5e845ea6c8329 (table default.Table_eb8f4b9d6a8d4c9891bff8243880751d [id=d207f6aad69f4e269854886a771760c9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 45e39a7b58ac402da7c5e845ea6c8329 consensus state has no leader
I20260810 23:32:59.000094 12806 raft_consensus.cc:2454] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
W20260810 23:32:58.949215 12700 leader_election.cc:341] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:59.003604 12755 tablet_service.cc:1467] Processing CreateTablet for tablet 0e8120281fe84e9a89f25d07c1d7b6a8 (DEFAULT_TABLE table=default.Table_a7d468b7f476404c8b400f3923cd1543 [id=bc138fc988a942b093d18a66e0cf605c]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:59.007623 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 58453.3us
I20260810 23:32:59.008970 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e8120281fe84e9a89f25d07c1d7b6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:59.011714 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 0e8120281fe84e9a89f25d07c1d7b6a8 (DEFAULT_TABLE table=default.Table_a7d468b7f476404c8b400f3923cd1543 [id=bc138fc988a942b093d18a66e0cf605c]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:59.009274 12698 leader_election.cc:341] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:59.018100 12698 leader_election.cc:302] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:32:59.020485 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 53578.9us
I20260810 23:32:59.023836 13023 tablet_service.cc:1467] Processing CreateTablet for tablet 09bdc7c2aace45da9fce81c4904db5d4 (DEFAULT_TABLE table=default.Table_2e4541bbe3aa45c1a1c85533c2ed79f3 [id=c9327e173e574f75b4886311e0933acc]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:59.031418 12215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111) [suppressed 3 similar messages]
I20260810 23:32:58.871966 12744 rpcz_store.cc:276] 0810 23:32:57.788779 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.868399 (+ 79620us) service_pool.cc:227] Handling call
0810 23:32:58.870596 (+1002197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:32:59.044291 15198 raft_consensus.cc:2726] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:32:59.073431 12229 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet c01ede4b52c3498a924340b39c3c2435 (table default.Table_ebe848b2f9ba4077a556369c87aa436a [id=ae7b5faec4c34d35981e0a1d464202ef]). Expected version 1 got 0
I20260810 23:32:59.076663 12229 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:59.083557 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:32:59.084636 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet c01ede4b52c3498a924340b39c3c2435 (table default.Table_ebe848b2f9ba4077a556369c87aa436a [id=ae7b5faec4c34d35981e0a1d464202ef]). Expected version 1 got 0
I20260810 23:32:59.085716 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:59.119036 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9443fb32d79941d58b13607e8ac064cd" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:32:59.134498 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 7898730d71214560b726ed6a156f04b0 (DEFAULT_TABLE table=default.Table_e1ff8765e6e547fda12a138edb0a5bb1 [id=c0859eb309eb48f2a1976147213b0aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:59.140214 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 7898730d71214560b726ed6a156f04b0 (DEFAULT_TABLE table=default.Table_e1ff8765e6e547fda12a138edb0a5bb1 [id=c0859eb309eb48f2a1976147213b0aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:59.141695 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7898730d71214560b726ed6a156f04b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:59.143293 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7898730d71214560b726ed6a156f04b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:59.149118 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5]). Expected version 2 got 0
I20260810 23:32:59.149919 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (table default.Table_34a45145441d42cbbaf01a336059d4ec [id=2c8320255aab4036b5350d082f9e04ef]). Expected version 1 got 0
I20260810 23:32:59.158432 13691 tablet_bootstrap.cc:492] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:32:59.159152 13691 ts_tablet_manager.cc:1397] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.801s user 0.015s sys 0.006s
W20260810 23:32:59.090523 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 59142us
I20260810 23:32:59.166044 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 7898730d71214560b726ed6a156f04b0 (DEFAULT_TABLE table=default.Table_e1ff8765e6e547fda12a138edb0a5bb1 [id=c0859eb309eb48f2a1976147213b0aab]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:32:58.979370 12234 rpcz_store.cc:269] 0810 23:32:57.779916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.785726 (+ 5810us) service_pool.cc:227] Handling call
0810 23:32:58.978782 (+1193056us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:58.549863 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:58.550748 (+ 885us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:58.550799 (+ 51us) write_op.cc:435] Acquired schema lock
0810 23:32:58.550822 (+ 23us) tablet.cc:660] Decoding operations
0810 23:32:58.551505 (+ 683us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:58.551580 (+ 75us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:58.551661 (+ 81us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:58.552492 (+ 831us) tablet.cc:699] Row locks acquired
0810 23:32:58.552569 (+ 77us) write_op.cc:260] PREPARE: finished
0810 23:32:58.552821 (+ 252us) write_op.cc:270] Start()
0810 23:32:58.552959 (+ 138us) write_op.cc:276] Timestamp: P: 1786404778552740 usec, L: 0
0810 23:32:58.552981 (+ 22us) op_driver.cc:331] REPLICATION: starting
0810 23:32:58.617189 (+ 64208us) log.cc:842] Serialized 919 byte log entry
0810 23:32:58.619782 (+ 2593us) spinlock_profiling.cc:243] Waited 63.1 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:58.649617 (+ 29835us) op_driver.cc:444] REPLICATION: finished
0810 23:32:58.679516 (+ 29899us) spinlock_profiling.cc:243] Waited 27 ms on lock 0x7b60000801c8. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce76ac361 00007f0ce781280c 00007f0ce77753af 00007f0ce7775914 00007f0ce7777a78 00007f0ce7777a28 00007f0ce77779a8 00007f0ce7777963 00007f0ce7776c4c 00007f0ce929a206 00007f0ce9292808
0810 23:32:58.707887 (+ 28371us) write_op.cc:301] APPLY: starting
0810 23:32:58.708003 (+ 116us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:58.708206 (+ 203us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:58.708243 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:58.708860 (+ 617us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:58.709150 (+ 290us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:58.709239 (+ 89us) write_op.cc:312] APPLY: finished
0810 23:32:58.709626 (+ 387us) log.cc:842] Serialized 29 byte log entry
0810 23:32:58.812754 (+103128us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:58.812889 (+ 135us) write_op.cc:454] Released schema lock
0810 23:32:58.812950 (+ 61us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:58.852750 (+ 39800us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:58.852790 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":10262,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":28098,"apply.run_cpu_time_us":3089,"apply.run_wall_time_us":145205,"mutex_wait_us":24799,"num_ops":2,"prepare.queue_time_us":245,"prepare.run_cpu_time_us":6824,"prepare.run_wall_time_us":78639,"raft.queue_time_us":134,"raft.run_cpu_time_us":1384,"raft.run_wall_time_us":2729,"replication_time_us":96500,"spinlock_wait_cycles":207079936}]]}
I20260810 23:32:59.169965 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_c764bc5f83fe4b25aaff52cac5e1009e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:59.170980 15082 ts_tablet_manager.cc:1918] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:59.172223 15082 log.cc:1196] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/4d0d9227d8574de8bd73041f3e8043a0
I20260810 23:32:59.173770 15082 ts_tablet_manager.cc:1939] T 4d0d9227d8574de8bd73041f3e8043a0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:59.173717 13691 raft_consensus.cc:348] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.174577 13691 raft_consensus.cc:374] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:59.174821 13691 raft_consensus.cc:729] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:32:59.182199 15156 raft_consensus.cc:482] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:59.182730 15156 raft_consensus.cc:504] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.183038 13274 tablet_service.cc:1467] Processing CreateTablet for tablet 0e8120281fe84e9a89f25d07c1d7b6a8 (DEFAULT_TABLE table=default.Table_a7d468b7f476404c8b400f3923cd1543 [id=bc138fc988a942b093d18a66e0cf605c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:59.184916 13274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e8120281fe84e9a89f25d07c1d7b6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:59.186172 15156 leader_election.cc:288] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:59.175624 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9443fb32d79941d58b13607e8ac064cd" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:59.196038 12958 consensus_peers.cc:487] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:59.175614 13691 consensus_queue.cc:260] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.211969 13665 tablet_bootstrap.cc:492] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:59.212960 13665 ts_tablet_manager.cc:1397] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.295s user 0.012s sys 0.014s
I20260810 23:32:59.187873 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e8311c941b43058384cee6d8263dc1" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
W20260810 23:32:59.202648 12698 leader_election.cc:341] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:59.223587 12805 raft_consensus.cc:2454] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:59.216784 13691 ts_tablet_manager.cc:1428] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.057s user 0.004s sys 0.003s
I20260810 23:32:59.225395 13691 tablet_bootstrap.cc:492] T 9443fb32d79941d58b13607e8ac064cd P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:32:59.231082 13213 leader_election.cc:302] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
W20260810 23:32:59.204813 15206 log.cc:924] Time spent T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
W20260810 23:32:59.201041 12215 consensus_peers.cc:487] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 -> Peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Couldn't send request to peer 198f534261994047a68e1d509bb2fd8b. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260810 23:32:59.235702 12223 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet c01ede4b52c3498a924340b39c3c2435 (table default.Table_ebe848b2f9ba4077a556369c87aa436a [id=ae7b5faec4c34d35981e0a1d464202ef]). Expected version 1 got 0
I20260810 23:32:59.248598 12223 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:59.253947 13665 raft_consensus.cc:348] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.254647 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09bdc7c2aace45da9fce81c4904db5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:59.254858 13665 raft_consensus.cc:374] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:59.255108 13665 raft_consensus.cc:729] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:59.255927 13665 consensus_queue.cc:260] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.257284 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e8120281fe84e9a89f25d07c1d7b6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:59.239248 15217 raft_consensus.cc:2781] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:59.258307 15217 raft_consensus.cc:482] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:59.258771 15217 raft_consensus.cc:3037] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:59.268648 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7898730d71214560b726ed6a156f04b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:59.282545 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 270) took 1734 ms. Trace:
I20260810 23:32:59.283123 15198 raft_consensus.cc:482] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:59.283794 15198 raft_consensus.cc:504] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.289021 13665 ts_tablet_manager.cc:1428] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.075s user 0.006s sys 0.002s
I20260810 23:32:59.291038 15198 leader_election.cc:288] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:59.291993 15082 rpcz_store.cc:276] 0810 23:32:57.547997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.550847 (+ 2850us) service_pool.cc:227] Handling call
0810 23:32:58.577660 (+1026813us) ts_tablet_manager.cc:1145] Deleting tablet 4d0d9227d8574de8bd73041f3e8043a0
0810 23:32:58.577731 (+ 71us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:59.038940 (+461209us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:59.155890 (+116950us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:59.282412 (+126522us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54712,"tablet-delete.queue_time_us":1025907}
I20260810 23:32:59.293938 15082 tablet_replica.cc:330] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:59.295188 15082 raft_consensus.cc:2229] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:59.295888 15082 pending_rounds.cc:62] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: Trying to abort 1 pending ops.
I20260810 23:32:59.295742 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 4d0d9227d8574de8bd73041f3e8043a0 (table default.Table_bf4e391ee20b45d393a269203b401bc0 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]) successfully deleted
W20260810 23:32:59.297902 15206 consensus_queue.cc:1159] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "191d120f4c7f44fc8475305a6d2e75f3" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20260810 23:32:59.300415 14121 ts_tablet_manager.cc:1918] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260810 23:32:59.242542 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c764bc5f83fe4b25aaff52cac5e1009e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:59.308580 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7bd9b6ca4845d8b8705ff970c8f06e" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:59.310515 12698 leader_election.cc:341] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:59.300884 15133 raft_consensus.cc:482] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:59.312526 15133 raft_consensus.cc:504] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.314673 15133 leader_election.cc:288] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:59.317425 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e8311c941b43058384cee6d8263dc1" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:59.300446 13691 tablet_bootstrap.cc:654] T 9443fb32d79941d58b13607e8ac064cd P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:59.326294 14121 log.cc:1196] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/9c29b11192604ac18c21b7bdc0238fb3
I20260810 23:32:59.327224 14121 ts_tablet_manager.cc:1939] T 9c29b11192604ac18c21b7bdc0238fb3 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:32:59.329800 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 224) took 7081 ms. Trace:
I20260810 23:32:59.330121 14121 rpcz_store.cc:276] 0810 23:32:52.248569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.249694 (+ 1125us) service_pool.cc:227] Handling call
0810 23:32:58.231012 (+5981318us) ts_tablet_manager.cc:1145] Deleting tablet 9c29b11192604ac18c21b7bdc0238fb3
0810 23:32:58.231060 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:58.756339 (+525279us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:59.300077 (+543738us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:59.329735 (+ 29658us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5980680}
I20260810 23:32:59.331324 13549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:59.331676 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 9c29b11192604ac18c21b7bdc0238fb3 (table default.Table_dd2cd30e5e2349de9180be01866dc513 [id=8754da7be72b4eef80daacdcdfde82a7]) successfully deleted
W20260810 23:32:59.344341 12952 consensus_peers.cc:487] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3d4c879ca6a402dbafe91940e7b367a. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:59.344503 13665 tablet_bootstrap.cc:492] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:59.346679 13549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:59.347661 13549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:59.327472 15217 raft_consensus.cc:504] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.351706 13335 raft_consensus.cc:2379] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:32:59.359845 15217 leader_election.cc:288] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:59.366887 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e8311c941b43058384cee6d8263dc1" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:32:59.367566 12805 raft_consensus.cc:3037] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:59.376247 12805 raft_consensus.cc:2454] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
W20260810 23:32:59.364931 13213 consensus_peers.cc:487] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:59.389591 15227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e964330cf113403a8aa5b73ddd3b31d0)
I20260810 23:32:59.390288 15227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:32:59.303539 15082 pending_rounds.cc:68] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317113966763679744 op_type: NO_OP noop_request { }
I20260810 23:32:59.393558 12236 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet c01ede4b52c3498a924340b39c3c2435 (table default.Table_ebe848b2f9ba4077a556369c87aa436a [id=ae7b5faec4c34d35981e0a1d464202ef]). Expected version 1 got 0
I20260810 23:32:59.393985 15082 raft_consensus.cc:2866] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:32:59.395898 12236 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:59.396404 15082 raft_consensus.cc:2258] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:59.403955 15227 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:32:59.406612 15082 ts_tablet_manager.cc:1905] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:59.413916 13514 tablet_bootstrap.cc:492] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:59.423552 13213 leader_election.cc:302] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:59.424908 15217 raft_consensus.cc:2781] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:59.425518 15217 raft_consensus.cc:686] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:32:59.425860 13514 ts_tablet_manager.cc:1397] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.857s user 0.036s sys 0.048s
I20260810 23:32:59.426630 15217 consensus_queue.cc:237] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.428280 15156 consensus_queue.cc:1035] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:59.435209 12226 catalog_manager.cc:5495] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:59.444926 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet c01ede4b52c3498a924340b39c3c2435 (table default.Table_ebe848b2f9ba4077a556369c87aa436a [id=ae7b5faec4c34d35981e0a1d464202ef]). Expected version 1 got 0
I20260810 23:32:59.454331 15234 tablet.cc:1722] T e6b8bf38da5647889756935404caad75 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:59.465729 15225 tablet.cc:1722] T e6b8bf38da5647889756935404caad75 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:32:59.482980 13691 tablet_bootstrap.cc:492] T 9443fb32d79941d58b13607e8ac064cd P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
W20260810 23:32:59.493582 12454 leader_election.cc:334] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101): Network error: Client connection negotiation failed: client connection to 127.11.226.189:42101: connect: Connection refused (error 111)
I20260810 23:32:59.495285 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 354 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:32:59.497424 13665 tablet_bootstrap.cc:654] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:59.497704 12240 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 323) took 3524 ms. Trace:
I20260810 23:32:59.498049 12240 rpcz_store.cc:276] 0810 23:32:55.973218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:58.001934 (+2028716us) service_pool.cc:227] Handling call
0810 23:32:58.036493 (+ 34559us) spinlock_profiling.cc:243] Waited 32.4 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e77d6 00007f0ce6e2d634 00007f0ce6e2d5c2 00007f0ce6e2d4f1 00007f0ce6e2d46c 00007f0ce6e2c722 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:58.064758 (+ 28265us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:58.066172 (+ 1414us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:58.066840 (+ 668us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:58.067089 (+ 249us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:58.190469 (+123380us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:59.497619 (+1307150us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:58.118629 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:58.119775 (+ 1146us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:58.119830 (+ 55us) write_op.cc:435] Acquired schema lock
0810 23:32:58.119844 (+ 14us) tablet.cc:660] Decoding operations
0810 23:32:58.120571 (+ 727us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:58.120729 (+ 158us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:58.120791 (+ 62us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:58.121161 (+ 370us) tablet.cc:699] Row locks acquired
0810 23:32:58.121173 (+ 12us) write_op.cc:260] PREPARE: finished
0810 23:32:58.121441 (+ 268us) write_op.cc:270] Start()
0810 23:32:58.121630 (+ 189us) write_op.cc:276] Timestamp: P: 1786404778121355 usec, L: 0
0810 23:32:58.121647 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:32:58.163821 (+ 42174us) log.cc:842] Serialized 405 byte log entry
0810 23:32:58.178792 (+ 14971us) spinlock_profiling.cc:243] Waited 41.4 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:58.188300 (+ 9508us) op_driver.cc:444] REPLICATION: finished
0810 23:32:58.188706 (+ 406us) write_op.cc:301] APPLY: starting
0810 23:32:58.188772 (+ 66us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:58.188877 (+ 105us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:58.188897 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:58.189178 (+ 281us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:58.189325 (+ 147us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:58.189370 (+ 45us) write_op.cc:312] APPLY: finished
0810 23:32:58.189591 (+ 221us) log.cc:842] Serialized 23 byte log entry
0810 23:32:58.189760 (+ 169us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:58.189835 (+ 75us) write_op.cc:454] Released schema lock
0810 23:32:58.189866 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:58.190081 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:58.190106 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":74504960,"child_traces":[["op",{"apply.queue_time_us":227,"apply.run_cpu_time_us":1574,"apply.run_wall_time_us":1574,"num_ops":1,"prepare.queue_time_us":44602,"prepare.run_cpu_time_us":6743,"prepare.run_wall_time_us":76259,"raft.queue_time_us":152,"raft.run_cpu_time_us":1082,"raft.run_wall_time_us":1536,"replication_time_us":66526,"spinlock_wait_cycles":95309696}]]}
I20260810 23:32:59.507817 12456 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 681674cb24274f10b200f264f155c2d1; no voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:32:59.509070 15227 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260810 23:32:59.530922 12240 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_1ec6152b7f994da6987cb0683216ade9" } new_table_name: "default.Table_9a58bbb30a414d8b8ec6893b106ee0a1" modify_external_catalogs: true
I20260810 23:32:59.548458 13514 raft_consensus.cc:348] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:59.549347 13514 raft_consensus.cc:374] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:59.549772 13514 raft_consensus.cc:729] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:59.512679 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:32:59.560166 12235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet f935119e4d9b4accb79db98185c80aca
I20260810 23:32:59.561920 12235 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 324) took 2661 ms. Trace:
I20260810 23:32:59.570680 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet f935119e4d9b4accb79db98185c80aca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:58 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:59.559908 13514 consensus_queue.cc:260] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:59.587387 15204 raft_consensus.cc:482] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:59.562263 12235 rpcz_store.cc:276] 0810 23:32:56.900351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:58.086958 (+1186607us) service_pool.cc:227] Handling call
0810 23:32:58.095136 (+ 8178us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:58.149994 (+ 54858us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:58.150149 (+ 155us) catalog_manager.cc:2761] Locking tablets
0810 23:32:58.150787 (+ 638us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:58.416668 (+265881us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:58.417054 (+ 386us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:59.560011 (+1142957us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:59.561848 (+ 1837us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:58.220270 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:58.220798 (+ 528us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:58.220829 (+ 31us) write_op.cc:435] Acquired schema lock
0810 23:32:58.220839 (+ 10us) tablet.cc:660] Decoding operations
0810 23:32:58.221288 (+ 449us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:58.221334 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:58.221359 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:58.221710 (+ 351us) tablet.cc:699] Row locks acquired
0810 23:32:58.221723 (+ 13us) write_op.cc:260] PREPARE: finished
0810 23:32:58.222021 (+ 298us) write_op.cc:270] Start()
0810 23:32:58.222153 (+ 132us) write_op.cc:276] Timestamp: P: 1786404778221807 usec, L: 0
0810 23:32:58.222168 (+ 15us) op_driver.cc:331] REPLICATION: starting
0810 23:32:58.251504 (+ 29336us) log.cc:842] Serialized 774 byte log entry
0810 23:32:58.402189 (+150685us) op_driver.cc:444] REPLICATION: finished
0810 23:32:58.407889 (+ 5700us) write_op.cc:301] APPLY: starting
0810 23:32:58.408005 (+ 116us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:58.408212 (+ 207us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:58.408326 (+ 114us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:58.408929 (+ 603us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:58.409218 (+ 289us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:58.409286 (+ 68us) write_op.cc:312] APPLY: finished
0810 23:32:58.409612 (+ 326us) log.cc:842] Serialized 29 byte log entry
0810 23:32:58.410531 (+ 919us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:58.410647 (+ 116us) write_op.cc:454] Released schema lock
0810 23:32:58.410695 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:58.417243 (+ 6548us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:58.417278 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5386,"apply.run_cpu_time_us":3006,"apply.run_wall_time_us":9790,"num_ops":2,"prepare.queue_time_us":64700,"prepare.run_cpu_time_us":3465,"prepare.run_wall_time_us":34445,"replication_time_us":179913}]]}
I20260810 23:32:59.588186 15204 raft_consensus.cc:504] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.593065 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41ddf9f6f71443ba75c97d2b010372c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:32:59.594334 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet f935119e4d9b4accb79db98185c80aca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:58 UTC) from {username='slave'} at 127.0.0.1:47546
W20260810 23:32:59.594645 12698 leader_election.cc:341] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:59.595988 15204 leader_election.cc:288] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:59.615581 15234 tablet.cc:1722] T 217a7582902244f9b0ca65be1c20af22 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:32:59.617764 12228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 10033af79e854103bc47a046d2219ca0
I20260810 23:32:59.621039 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 10033af79e854103bc47a046d2219ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:58 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:32:59.619756 12228 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 326) took 2062 ms. Trace:
I20260810 23:32:59.624254 12228 rpcz_store.cc:276] 0810 23:32:57.557013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:58.092411 (+535398us) service_pool.cc:227] Handling call
0810 23:32:58.094864 (+ 2453us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:58.146027 (+ 51163us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:32:58.146201 (+ 174us) catalog_manager.cc:2761] Locking tablets
0810 23:32:58.146800 (+ 599us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:32:58.373079 (+226279us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:32:58.373534 (+ 455us) catalog_manager.cc:2809] Committing in-memory state
0810 23:32:59.617615 (+1244081us) catalog_manager.cc:2813] Aborting table tasks
0810 23:32:59.619686 (+ 2071us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:58.195000 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:58.195555 (+ 555us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:58.195586 (+ 31us) write_op.cc:435] Acquired schema lock
0810 23:32:58.195603 (+ 17us) tablet.cc:660] Decoding operations
0810 23:32:58.196045 (+ 442us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:58.196091 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:58.196123 (+ 32us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:58.196572 (+ 449us) tablet.cc:699] Row locks acquired
0810 23:32:58.196586 (+ 14us) write_op.cc:260] PREPARE: finished
0810 23:32:58.196750 (+ 164us) write_op.cc:270] Start()
0810 23:32:58.196852 (+ 102us) write_op.cc:276] Timestamp: P: 1786404778196683 usec, L: 0
0810 23:32:58.196872 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:32:58.208857 (+ 11985us) log.cc:842] Serialized 768 byte log entry
0810 23:32:58.212852 (+ 3995us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:32:58.368521 (+155669us) op_driver.cc:444] REPLICATION: finished
0810 23:32:58.369048 (+ 527us) write_op.cc:301] APPLY: starting
0810 23:32:58.369169 (+ 121us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:58.369367 (+ 198us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:58.369417 (+ 50us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:58.369872 (+ 455us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:58.370126 (+ 254us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:32:58.370181 (+ 55us) write_op.cc:312] APPLY: finished
0810 23:32:58.370490 (+ 309us) log.cc:842] Serialized 29 byte log entry
0810 23:32:58.370810 (+ 320us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:58.370956 (+ 146us) write_op.cc:454] Released schema lock
0810 23:32:58.371020 (+ 64us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:58.371389 (+ 369us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:58.371427 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":228,"apply.run_cpu_time_us":2730,"apply.run_wall_time_us":2729,"num_ops":2,"prepare.queue_time_us":40601,"prepare.run_cpu_time_us":3979,"prepare.run_wall_time_us":25023,"raft.queue_time_us":3583,"raft.run_cpu_time_us":945,"raft.run_wall_time_us":1390,"replication_time_us":171484,"spinlock_wait_cycles":25705728}]]}
I20260810 23:32:59.636590 13273 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c29b11192604ac18c21b7bdc0238fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:38 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:59.643184 13514 ts_tablet_manager.cc:1428] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.217s user 0.007s sys 0.003s
I20260810 23:32:59.644661 13514 tablet_bootstrap.cc:492] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:59.654217 13514 tablet_bootstrap.cc:654] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:59.649188 13273 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0dd19112504f8ca57f64dcf0bd50a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:59.661516 15253 tablet.cc:1722] T 217a7582902244f9b0ca65be1c20af22 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:32:59.640894 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet 10033af79e854103bc47a046d2219ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:58 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:59.671623 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_e156205c664d4b45b29a0e258e39cf34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:59.673707 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e156205c664d4b45b29a0e258e39cf34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:59.690775 13665 tablet_bootstrap.cc:492] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:59.693495 13514 tablet_bootstrap.cc:492] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:59.694341 13514 ts_tablet_manager.cc:1397] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.050s user 0.013s sys 0.005s
I20260810 23:32:59.698725 13514 raft_consensus.cc:348] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:59.703961 13665 ts_tablet_manager.cc:1397] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.360s user 0.008s sys 0.011s
I20260810 23:32:59.709488 13514 raft_consensus.cc:374] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:59.711483 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b81e7056a441d2854ba5ab6b4c5433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:55 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:59.713253 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d4c879ca6a402dbafe91940e7b367a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:39 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:59.714861 13514 raft_consensus.cc:729] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:59.715107 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdec13ee70d402fa3a08119e7bf1313 with delete_type TABLET_DATA_DELETED (replaced by tablet d603569c54ef4ffba9941d0cc3e0f97f) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:59.720949 15225 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:47546 (request call id 277) took 1363 ms. Trace:
I20260810 23:32:59.708256 13665 raft_consensus.cc:348] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:59.727011 13665 raft_consensus.cc:374] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:59.727416 13665 raft_consensus.cc:729] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:59.728287 13665 consensus_queue.cc:260] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:32:59.716501 13514 consensus_queue.cc:260] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:59.731412 13514 ts_tablet_manager.cc:1428] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.037s user 0.003s sys 0.004s
I20260810 23:32:59.732709 13514 tablet_bootstrap.cc:492] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:59.714388 15082 ts_tablet_manager.cc:1918] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:32:59.736466 15082 log.cc:1196] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/f47b8b57d5b44a018bc2d7b3f976d803
I20260810 23:32:59.737680 15082 ts_tablet_manager.cc:1939] T f47b8b57d5b44a018bc2d7b3f976d803 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:59.727916 15225 rpcz_store.cc:276] 0810 23:32:58.357472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:58.357655 (+ 183us) service_pool.cc:227] Handling call
0810 23:32:59.720800 (+1363145us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:58.527673 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 23:32:58.528486 (+ 813us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 23:32:58.528576 (+ 90us) alter_schema_op.cc:65] Acquired schema lock
0810 23:32:58.528856 (+ 280us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 23:32:58.529034 (+ 178us) alter_schema_op.cc:126] START. Timestamp: 1786404778528933
0810 23:32:58.529105 (+ 71us) op_driver.cc:331] REPLICATION: starting
0810 23:32:58.604403 (+ 75298us) log.cc:842] Serialized 173 byte log entry
0810 23:32:59.320998 (+716595us) op_driver.cc:444] REPLICATION: finished
0810 23:32:59.465606 (+144608us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 23:32:59.475483 (+ 9877us) log.cc:842] Serialized 5 byte log entry
0810 23:32:59.683594 (+208111us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:59.711227 (+ 27633us) log.cc:842] Serialized 14 byte log entry
0810 23:32:59.720388 (+ 9161us) alter_schema_op.cc:71] Released schema lock
0810 23:32:59.720487 (+ 99us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":144337,"prepare.queue_time_us":48123,"prepare.run_cpu_time_us":4353,"prepare.run_wall_time_us":128817,"replication_time_us":791777,"thread_start_us":263825,"threads_started":3,"wal-append.queue_time_us":10781}]]}
I20260810 23:32:59.745419 13665 ts_tablet_manager.cc:1428] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.041s user 0.008s sys 0.000s
W20260810 23:32:59.746066 12215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:42101, user_credentials={real_user=slave}} blocked reactor thread for 122931us
I20260810 23:32:59.746917 13665 tablet_bootstrap.cc:492] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:59.749785 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 271) took 2126 ms. Trace:
I20260810 23:32:59.759176 15082 rpcz_store.cc:276] 0810 23:32:57.623180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.623430 (+ 250us) service_pool.cc:227] Handling call
0810 23:32:59.293733 (+1670303us) ts_tablet_manager.cc:1145] Deleting tablet f47b8b57d5b44a018bc2d7b3f976d803
0810 23:32:59.293790 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:59.651235 (+357445us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:59.714024 (+ 62789us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:59.749671 (+ 35647us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1669378}
I20260810 23:32:59.763121 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet f47b8b57d5b44a018bc2d7b3f976d803 (table default.Table_7ba7fd44e98542e78c604739a5086a91 [id=a7a55f9e68f04a9c9c43b9607cd901c6]) successfully deleted
I20260810 23:32:59.764783 15216 consensus_queue.cc:1035] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:32:59.768287 15082 tablet_replica.cc:330] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:59.769531 15082 raft_consensus.cc:2229] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:59.770278 15082 raft_consensus.cc:2258] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:59.772939 13514 tablet_bootstrap.cc:654] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:59.775269 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet f47b8b57d5b44a018bc2d7b3f976d803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:59.776342 12229 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 35) took 2315 ms (client timeout 1000 ms). Trace:
W20260810 23:32:59.776773 12229 rpcz_store.cc:269] 0810 23:32:57.461244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.564226 (+102982us) service_pool.cc:227] Handling call
0810 23:32:59.776234 (+2212008us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67572,"reported_tablets":19}
I20260810 23:32:59.778203 12229 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_1ebd777d240743bbb5739b63d7e00b39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:32:59.780324 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 423e8d583d704fa9834026ebcd57e17e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:32:59.782035 12224 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 41) took 2096 ms (client timeout 1000 ms). Trace:
W20260810 23:32:59.782405 12224 rpcz_store.cc:269] 0810 23:32:57.685329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.762683 (+ 77354us) service_pool.cc:227] Handling call
0810 23:32:59.781950 (+2019267us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54016,"reported_tablets":24}
I20260810 23:32:59.782852 13665 tablet_bootstrap.cc:654] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:59.784338 12224 master_service.cc:946] ReplaceTablet: received request to replace tablet 422b25457d1549e186641fcedd8feab9 from {username='slave'} at 127.0.0.1:34742
I20260810 23:32:59.785092 15082 ts_tablet_manager.cc:1905] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:59.788316 15250 tablet_replica.cc:330] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:32:59.800910 15250 raft_consensus.cc:2229] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:59.801600 13514 tablet_bootstrap.cc:492] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:32:59.802145 13514 ts_tablet_manager.cc:1397] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.003s
W20260810 23:32:59.809800 12236 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 34) took 3418 ms (client timeout 1000 ms). Trace:
W20260810 23:32:59.814126 12236 rpcz_store.cc:269] 0810 23:32:56.391552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.322017 (+930465us) service_pool.cc:227] Handling call
0810 23:32:59.809708 (+2487691us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":168475,"reported_tablets":18,"spinlock_wait_cycles":139648}
I20260810 23:32:59.809280 15216 raft_consensus.cc:482] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:59.821197 15216 raft_consensus.cc:504] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.825191 13514 raft_consensus.cc:348] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:59.826131 13514 raft_consensus.cc:374] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:59.826462 13514 raft_consensus.cc:729] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:32:59.823809 15216 leader_election.cc:288] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:59.829185 15250 raft_consensus.cc:2258] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:59.834350 13514 consensus_queue.cc:260] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:59.836663 15187 raft_consensus.cc:482] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:59.837272 15187 raft_consensus.cc:504] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:59.838526 15250 ts_tablet_manager.cc:1905] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:59.841493 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975a9bab969e46d2b939fc76ce350b99" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:59.843330 12698 leader_election.cc:341] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:59.844304 15187 leader_election.cc:288] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:59.849121 15082 ts_tablet_manager.cc:1918] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:32:59.849756 15082 log.cc:1196] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/a41ddf9f6f71443ba75c97d2b010372c
I20260810 23:32:59.850982 15082 ts_tablet_manager.cc:1939] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:32:59.854290 13514 ts_tablet_manager.cc:1428] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.052s user 0.009s sys 0.000s
I20260810 23:32:59.860433 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 273) took 2200 ms. Trace:
I20260810 23:32:59.861068 15082 rpcz_store.cc:276] 0810 23:32:57.660396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.661472 (+ 1076us) service_pool.cc:227] Handling call
0810 23:32:59.766242 (+2104770us) ts_tablet_manager.cc:1145] Deleting tablet a41ddf9f6f71443ba75c97d2b010372c
0810 23:32:59.768097 (+ 1855us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:59.810997 (+ 42900us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:59.840393 (+ 29396us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:59.860340 (+ 19947us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2103701}
I20260810 23:32:59.862666 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet a41ddf9f6f71443ba75c97d2b010372c (table default.Table_73f2b1b207fb45f2adecbd37412c5e6f [id=c57f7c273a9841c1ac02523a733a383f]) successfully deleted
I20260810 23:32:59.863008 15082 tablet_replica.cc:330] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:32:59.864070 15082 raft_consensus.cc:2229] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:59.864727 15082 raft_consensus.cc:2258] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:59.865159 13514 tablet_bootstrap.cc:492] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:32:59.810199 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d697e7533af4fc0a15cb41f3fcce4db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:32:59.819859 13665 tablet_bootstrap.cc:492] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:59.875500 13665 ts_tablet_manager.cc:1397] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.129s user 0.013s sys 0.005s
I20260810 23:32:59.878582 13665 raft_consensus.cc:348] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.879361 13665 raft_consensus.cc:374] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:59.879606 13665 raft_consensus.cc:729] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:59.881011 13665 consensus_queue.cc:260] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.887070 13665 ts_tablet_manager.cc:1428] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20260810 23:32:59.891026 13665 tablet_bootstrap.cc:492] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:32:59.892513 15187 raft_consensus.cc:482] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:59.893159 15187 raft_consensus.cc:504] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:32:59.813513 12223 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 42) took 1346 ms (client timeout 1000 ms). Trace:
W20260810 23:32:59.896526 12223 rpcz_store.cc:269] 0810 23:32:58.467101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:58.567017 (+ 99916us) service_pool.cc:227] Handling call
0810 23:32:58.638305 (+ 71288us) spinlock_profiling.cc:243] Waited 35.2 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:59.813401 (+1175096us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":116564,"reported_tablets":23,"spinlock_wait_cycles":80845056}
I20260810 23:32:59.897143 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b9d20d09c4d1ba9a4b10959807e50" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:59.899060 12698 leader_election.cc:341] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:59.899994 15187 leader_election.cc:288] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:32:59.903445 13514 tablet_bootstrap.cc:654] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:59.905455 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9443fb32d79941d58b13607e8ac064cd" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:32:59.906191 12805 raft_consensus.cc:2454] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:32:59.913901 15082 ts_tablet_manager.cc:1905] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:59.934852 12234 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 330) took 1628 ms. Trace:
I20260810 23:32:59.935380 12234 rpcz_store.cc:276] 0810 23:32:58.306053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.168891 (+862838us) service_pool.cc:227] Handling call
0810 23:32:59.500122 (+331231us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:59.503235 (+ 3113us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:59.933925 (+430690us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:32:59.934379 (+ 454us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:32:59.934751 (+ 372us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:59.559287 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:59.572540 (+ 13253us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:59.572574 (+ 34us) write_op.cc:435] Acquired schema lock
0810 23:32:59.572595 (+ 21us) tablet.cc:660] Decoding operations
0810 23:32:59.573019 (+ 424us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:59.573115 (+ 96us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:59.573145 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:59.573414 (+ 269us) tablet.cc:699] Row locks acquired
0810 23:32:59.573434 (+ 20us) write_op.cc:260] PREPARE: finished
0810 23:32:59.573625 (+ 191us) write_op.cc:270] Start()
0810 23:32:59.573752 (+ 127us) write_op.cc:276] Timestamp: P: 1786404779573552 usec, L: 0
0810 23:32:59.573775 (+ 23us) op_driver.cc:331] REPLICATION: starting
0810 23:32:59.574509 (+ 734us) log.cc:842] Serialized 433 byte log entry
0810 23:32:59.869818 (+295309us) op_driver.cc:444] REPLICATION: finished
0810 23:32:59.898980 (+ 29162us) write_op.cc:301] APPLY: starting
0810 23:32:59.899060 (+ 80us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:59.899198 (+ 138us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:59.899316 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:59.899862 (+ 546us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:59.900095 (+ 233us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:32:59.900166 (+ 71us) write_op.cc:312] APPLY: finished
0810 23:32:59.900509 (+ 343us) log.cc:842] Serialized 29 byte log entry
0810 23:32:59.900717 (+ 208us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:59.900795 (+ 78us) write_op.cc:454] Released schema lock
0810 23:32:59.933382 (+ 32587us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":28927,"num_ops":2,"prepare.queue_time_us":30471,"prepare.run_cpu_time_us":3173,"prepare.run_wall_time_us":97137,"raft.queue_time_us":154,"raft.run_cpu_time_us":773,"raft.run_wall_time_us":772,"replication_time_us":295941,"spinlock_wait_cycles":51456,"thread_start_us":121627,"threads_started":2}]]}
I20260810 23:32:59.939136 13213 leader_election.cc:302] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:32:59.940121 15216 raft_consensus.cc:2781] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:59.940510 15216 raft_consensus.cc:482] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:59.940828 15216 raft_consensus.cc:3037] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:59.948727 12240 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc7adcb46ae0472384b3bb883ea58a74 consensus state has no leader
I20260810 23:32:59.950361 12240 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 331) took 1323 ms. Trace:
I20260810 23:32:59.951846 15216 raft_consensus.cc:504] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.953788 13665 tablet_bootstrap.cc:654] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:59.960512 15216 leader_election.cc:288] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:32:59.970453 12240 rpcz_store.cc:276] 0810 23:32:58.627416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.530407 (+902991us) service_pool.cc:227] Handling call
0810 23:32:59.556368 (+ 25961us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:32:59.616149 (+ 59781us) catalog_manager.cc:3616] Apply alter schema
0810 23:32:59.759328 (+143179us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:32:59.759612 (+ 284us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:32:59.914218 (+154606us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b5400031480. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce76ac361 00007f0ce777ca86 00007f0ce7772e03 00007f0ce7737ce7 00007f0ce7737af4 00007f0ce93181f1 00007f0ce9318a16 00007f0ce91d63a4 00007f0ce91cc81d 00007f0ce92e7812 00007f0ce6e2d634
0810 23:32:59.946376 (+ 32158us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:32:59.950280 (+ 3904us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:59.914657 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:59.915321 (+ 664us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:59.915359 (+ 38us) write_op.cc:435] Acquired schema lock
0810 23:32:59.915382 (+ 23us) tablet.cc:660] Decoding operations
0810 23:32:59.915836 (+ 454us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:59.915904 (+ 68us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:59.915951 (+ 47us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:59.916207 (+ 256us) tablet.cc:699] Row locks acquired
0810 23:32:59.916224 (+ 17us) write_op.cc:260] PREPARE: finished
0810 23:32:59.916414 (+ 190us) write_op.cc:270] Start()
0810 23:32:59.916690 (+ 276us) write_op.cc:276] Timestamp: P: 1786404779916345 usec, L: 0
0810 23:32:59.916714 (+ 24us) op_driver.cc:331] REPLICATION: starting
0810 23:32:59.917566 (+ 852us) log.cc:842] Serialized 405 byte log entry
0810 23:32:59.936758 (+ 19192us) op_driver.cc:444] REPLICATION: finished
0810 23:32:59.944525 (+ 7767us) write_op.cc:301] APPLY: starting
0810 23:32:59.944598 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:59.944709 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:59.944814 (+ 105us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:59.945055 (+ 241us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:59.945213 (+ 158us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:59.945269 (+ 56us) write_op.cc:312] APPLY: finished
0810 23:32:59.945516 (+ 247us) log.cc:842] Serialized 23 byte log entry
0810 23:32:59.945706 (+ 190us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:59.945783 (+ 77us) write_op.cc:454] Released schema lock
0810 23:32:59.945818 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:59.946058 (+ 240us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:59.946087 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":254525824,"child_traces":[["op",{"apply.queue_time_us":7554,"apply.run_cpu_time_us":1760,"apply.run_wall_time_us":1759,"num_ops":1,"prepare.queue_time_us":158,"prepare.run_cpu_time_us":3294,"prepare.run_wall_time_us":4702,"raft.queue_time_us":105,"raft.run_cpu_time_us":17,"raft.run_wall_time_us":19,"replication_time_us":19923,"threads_started":1}]]}
I20260810 23:32:59.974035 15250 ts_tablet_manager.cc:1918] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:59.975399 15187 raft_consensus.cc:482] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:59.976140 15187 raft_consensus.cc:504] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:32:59.979466 15187 leader_election.cc:288] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:32:59.974812 15250 log.cc:1196] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/f935119e4d9b4accb79db98185c80aca
I20260810 23:32:59.981403 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806fe5162aef4d36ad5e2ac324047426" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:32:59.986294 12698 leader_election.cc:341] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:32:59.987820 15250 ts_tablet_manager.cc:1939] T f935119e4d9b4accb79db98185c80aca P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:32:59.989157 13665 tablet_bootstrap.cc:492] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:32:59.989770 13665 ts_tablet_manager.cc:1397] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.099s user 0.013s sys 0.000s
I20260810 23:32:59.993633 13665 raft_consensus.cc:348] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:32:59.996132 13665 raft_consensus.cc:374] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:59.996423 13665 raft_consensus.cc:729] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:32:59.998276 13665 consensus_queue.cc:260] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.004446 12785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9443fb32d79941d58b13607e8ac064cd" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:00.005293 12785 raft_consensus.cc:3037] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:59.962379 12234 master_service.cc:946] ReplaceTablet: received request to replace tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd from {username='slave'} at 127.0.0.1:34742
W20260810 23:32:59.779938 12229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ebd777d240743bbb5739b63d7e00b39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:59.993396 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]) successfully deleted
W20260810 23:33:00.010079 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 9c29b11192604ac18c21b7bdc0238fb3 because the tablet was not found. No further retry: Not found: Tablet not found: 9c29b11192604ac18c21b7bdc0238fb3
W20260810 23:33:00.010982 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet ca0dd19112504f8ca57f64dcf0bd50a8 because the tablet was not found. No further retry: Not found: Tablet not found: ca0dd19112504f8ca57f64dcf0bd50a8
I20260810 23:32:59.993700 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_4ad6c4068abe4c64a425bea91494c59a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:00.013242 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ad6c4068abe4c64a425bea91494c59a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:00.017516 12226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 36) took 1496 ms (client timeout 1000 ms). Trace:
I20260810 23:33:00.000273 12223 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_2c509bc8585b4ca68502ce3897fa18fa" } new_table_name: "default.Table_0bf7fdea09a840b9a1394935cc53d643" modify_external_catalogs: true
I20260810 23:33:00.028873 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3049bcdc6e43c795961c74f0d008f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:00.035686 13665 ts_tablet_manager.cc:1428] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.046s user 0.008s sys 0.000s
W20260810 23:33:00.036445 12218 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet dc7adcb46ae0472384b3bb883ea58a74 (table default.Table_9a58bbb30a414d8b8ec6893b106ee0a1 [id=6b7c9683647d4c2e94418e9f2dcaa41e], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet dc7adcb46ae0472384b3bb883ea58a74 consensus state has no leader
I20260810 23:33:00.036540 15250 tablet_replica.cc:330] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:00.037889 15250 raft_consensus.cc:2229] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:00.038664 15250 raft_consensus.cc:2258] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:00.045666 15216 raft_consensus.cc:482] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:00.046250 15216 raft_consensus.cc:504] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:00.048795 15216 leader_election.cc:288] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:00.052253 13665 tablet_bootstrap.cc:492] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:00.067233 15250 ts_tablet_manager.cc:1905] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:00.106137 13691 ts_tablet_manager.cc:1397] T 9443fb32d79941d58b13607e8ac064cd P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.881s user 0.009s sys 0.008s
I20260810 23:33:00.064783 13008 tablet_service.cc:1514] Processing DeleteTablet for tablet f935119e4d9b4accb79db98185c80aca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:58 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:00.065505 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdec13ee70d402fa3a08119e7bf1313 with delete_type TABLET_DATA_DELETED (replaced by tablet d603569c54ef4ffba9941d0cc3e0f97f) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:00.066740 13014 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d0d9227d8574de8bd73041f3e8043a0 with delete_type TABLET_DATA_DELETED (replaced by tablet fb8d482c6ee54e45a41201d98a5b60b4) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:00.067569 13007 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c29b11192604ac18c21b7bdc0238fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:38 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:00.068547 13012 tablet_service.cc:1514] Processing DeleteTablet for tablet f47b8b57d5b44a018bc2d7b3f976d803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:00.069840 13010 tablet_service.cc:1514] Processing DeleteTablet for tablet fd9d90e1ceab443dbec12b2c81b975e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:00.070771 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3049bcdc6e43c795961c74f0d008f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:58598
W20260810 23:33:00.075045 13023 consensus_meta.cc:287] T 09bdc7c2aace45da9fce81c4904db5d4 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.733s user 0.008s sys 0.006s
I20260810 23:33:00.080752 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet 423e8d583d704fa9834026ebcd57e17e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:32:59.484084 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 223) took 7236 ms. Trace:
I20260810 23:33:00.110397 14121 rpcz_store.cc:276] 0810 23:32:52.247759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.249570 (+ 1811us) service_pool.cc:227] Handling call
0810 23:32:59.483667 (+7234097us) ts_tablet_manager.cc:1145] Deleting tablet 440de7d9de8d4be9a5322e1fe3388b9b
0810 23:32:59.483731 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:59.483986 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7232821}
I20260810 23:33:00.089290 13031 tablet_service.cc:1514] Processing DeleteTablet for tablet 10033af79e854103bc47a046d2219ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:58 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:00.089600 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806fe5162aef4d36ad5e2ac324047426" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:00.090499 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975a9bab969e46d2b939fc76ce350b99" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:00.096464 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b9d20d09c4d1ba9a4b10959807e50" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:00.097694 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7bd9b6ca4845d8b8705ff970c8f06e" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:00.104485 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41ddf9f6f71443ba75c97d2b010372c" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:00.117606 12785 raft_consensus.cc:2454] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
W20260810 23:33:00.118826 12700 leader_election.cc:341] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:00.119104 15082 ts_tablet_manager.cc:1918] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:00.119521 12700 leader_election.cc:302] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:00.120689 15187 raft_consensus.cc:2726] T 9443fb32d79941d58b13607e8ac064cd P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:00.122838 13665 tablet_bootstrap.cc:654] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:00.125381 13213 leader_election.cc:302] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:00.126489 15216 raft_consensus.cc:2781] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:00.126904 15187 raft_consensus.cc:482] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:00.127466 15187 raft_consensus.cc:504] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:00.129778 15187 leader_election.cc:288] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:33:00.127066 15216 raft_consensus.cc:686] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:33:00.131316 12785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb2ee1c9b41478da001aa67d7a7ad3c" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:00.132073 12785 raft_consensus.cc:2454] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:00.133013 15216 consensus_queue.cc:237] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.141824 13514 tablet_bootstrap.cc:492] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:33:00.142400 13514 ts_tablet_manager.cc:1397] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.278s user 0.024s sys 0.010s
I20260810 23:33:00.144227 12785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e8311c941b43058384cee6d8263dc1" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:00.119547 15082 log.cc:1196] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/1b1f24e226b94ccf8ad5a656c80a5b3b
I20260810 23:33:00.145313 12785 raft_consensus.cc:2379] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:33:00.151744 13213 leader_election.cc:302] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:00.152878 15156 raft_consensus.cc:2781] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:00.153258 15156 raft_consensus.cc:482] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:00.153591 15156 raft_consensus.cc:3037] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:00.154757 13514 raft_consensus.cc:348] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:00.155823 13514 raft_consensus.cc:374] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:00.156167 13514 raft_consensus.cc:729] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:33:00.156949 13514 consensus_queue.cc:260] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:00.162571 13514 ts_tablet_manager.cc:1428] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20260810 23:33:00.164638 15082 ts_tablet_manager.cc:1939] T 1b1f24e226b94ccf8ad5a656c80a5b3b P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:33:00.168097 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a480093c481941f0bb35949c7606f37f" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:00.167248 13514 tablet_bootstrap.cc:492] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:33:00.175091 12698 leader_election.cc:341] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:00.180908 15156 raft_consensus.cc:504] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:00.186129 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 274) took 2525 ms. Trace:
I20260810 23:33:00.197646 15082 rpcz_store.cc:276] 0810 23:32:57.660835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.662806 (+ 1971us) service_pool.cc:227] Handling call
0810 23:32:59.862807 (+2200001us) ts_tablet_manager.cc:1145] Deleting tablet 1b1f24e226b94ccf8ad5a656c80a5b3b
0810 23:32:59.862864 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:32:59.965903 (+103039us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:00.108832 (+142929us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:00.186036 (+ 77204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2198817}
I20260810 23:33:00.199576 15082 tablet_replica.cc:330] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
W20260810 23:33:00.201052 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 440de7d9de8d4be9a5322e1fe3388b9b because the tablet was not found. No further retry: Not found: Tablet not found: 440de7d9de8d4be9a5322e1fe3388b9b
I20260810 23:33:00.201218 15082 raft_consensus.cc:2229] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:00.201911 15082 raft_consensus.cc:2258] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:00.201941 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 1b1f24e226b94ccf8ad5a656c80a5b3b (table default.Table_d942bf134f9849718bb9eb12c24806a1 [id=ff8088bdfc9f456683dd802d62ae67a6]) successfully deleted
I20260810 23:33:00.204548 15082 ts_tablet_manager.cc:1905] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:00.209885 13691 raft_consensus.cc:348] T 9443fb32d79941d58b13607e8ac064cd P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.211162 13691 raft_consensus.cc:374] T 9443fb32d79941d58b13607e8ac064cd P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:00.211491 13691 raft_consensus.cc:729] T 9443fb32d79941d58b13607e8ac064cd P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:00.212695 14121 tablet_replica.cc:330] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
W20260810 23:33:00.215606 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 94358us
I20260810 23:33:00.213743 13691 consensus_queue.cc:260] T 9443fb32d79941d58b13607e8ac064cd P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.218406 15156 leader_election.cc:288] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:00.219604 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb2ee1c9b41478da001aa67d7a7ad3c" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:00.220518 12805 raft_consensus.cc:3037] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:00.190457 13665 tablet_bootstrap.cc:492] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:00.232595 13665 ts_tablet_manager.cc:1397] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.187s user 0.011s sys 0.002s
I20260810 23:33:00.234051 13691 ts_tablet_manager.cc:1428] T 9443fb32d79941d58b13607e8ac064cd P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.127s user 0.007s sys 0.002s
W20260810 23:33:00.019254 12226 rpcz_store.cc:269] 0810 23:32:58.521096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:58.789861 (+268765us) service_pool.cc:227] Handling call
0810 23:33:00.017397 (+1227536us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:59.788337 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:59.788936 (+ 599us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:59.788971 (+ 35us) write_op.cc:435] Acquired schema lock
0810 23:32:59.788989 (+ 18us) tablet.cc:660] Decoding operations
0810 23:32:59.789358 (+ 369us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:59.789404 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:59.789448 (+ 44us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:32:59.789654 (+ 206us) tablet.cc:699] Row locks acquired
0810 23:32:59.789675 (+ 21us) write_op.cc:260] PREPARE: finished
0810 23:32:59.789841 (+ 166us) write_op.cc:270] Start()
0810 23:32:59.789985 (+ 144us) write_op.cc:276] Timestamp: P: 1786404779789758 usec, L: 0
0810 23:32:59.790009 (+ 24us) op_driver.cc:331] REPLICATION: starting
0810 23:32:59.790764 (+ 755us) log.cc:842] Serialized 536 byte log entry
0810 23:32:59.919087 (+128323us) op_driver.cc:444] REPLICATION: finished
0810 23:32:59.928934 (+ 9847us) write_op.cc:301] APPLY: starting
0810 23:32:59.929010 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0810 23:32:59.929116 (+ 106us) tablet.cc:1368] finished BulkCheckPresence
0810 23:32:59.929240 (+ 124us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:32:59.929523 (+ 283us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:32:59.929683 (+ 160us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:32:59.929729 (+ 46us) write_op.cc:312] APPLY: finished
0810 23:32:59.929983 (+ 254us) log.cc:842] Serialized 23 byte log entry
0810 23:32:59.937004 (+ 7021us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:59.937088 (+ 84us) write_op.cc:454] Released schema lock
0810 23:32:59.937131 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0810 23:32:59.956220 (+ 19089us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:32:59.956250 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":9605,"apply.run_cpu_time_us":2134,"apply.run_wall_time_us":27567,"num_ops":1,"prepare.queue_time_us":68214,"prepare.run_cpu_time_us":2720,"prepare.run_wall_time_us":2720,"replication_time_us":128978,"spinlock_wait_cycles":33664,"thread_start_us":10280,"threads_started":1}]]}
I20260810 23:33:00.242198 13691 tablet_bootstrap.cc:492] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:00.222005 15321 consensus_queue.cc:1035] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20260810 23:33:00.256289 12226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_2f4392eef1c74daa870d52081ffb3c90" } modify_external_catalogs: true
I20260810 23:33:00.257191 12226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_2f4392eef1c74daa870d52081ffb3c90" } modify_external_catalogs: true
I20260810 23:33:00.251458 13665 raft_consensus.cc:348] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.260210 13665 raft_consensus.cc:374] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:00.262223 13665 raft_consensus.cc:729] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:00.262856 12235 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 332) took 1490 ms. Trace:
I20260810 23:33:00.262945 13665 consensus_queue.cc:260] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.263173 12235 rpcz_store.cc:276] 0810 23:32:58.772076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.595450 (+823374us) service_pool.cc:227] Handling call
0810 23:32:59.671324 (+ 75874us) spinlock_profiling.cc:243] Waited 67.9 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7156 00007f0ce6e24034 00007f0ce6e23fc2 00007f0ce6e23ef1 00007f0ce6e23e6c 00007f0ce6e23122 00007f0ce50c694c 00007f0ce50c5d86
0810 23:32:59.674297 (+ 2973us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:32:59.797016 (+122719us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:32:59.915498 (+118482us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b5400031480. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce76ac361 00007f0ce777ca86 00007f0ce7772e03 00007f0ce7737ce7 00007f0ce7737af4 00007f0ce93181f1 00007f0ce9318a16 00007f0ce91c5785 00007f0ce92e718f 00007f0ce6e24034 00007f0ce6e23fc2
0810 23:33:00.261985 (+346487us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:33:00.262470 (+ 485us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:33:00.262783 (+ 313us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:59.919524 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:59.920269 (+ 745us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:59.920303 (+ 34us) write_op.cc:435] Acquired schema lock
0810 23:32:59.920324 (+ 21us) tablet.cc:660] Decoding operations
0810 23:32:59.920788 (+ 464us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:59.920856 (+ 68us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:59.920901 (+ 45us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:59.921271 (+ 370us) tablet.cc:699] Row locks acquired
0810 23:32:59.921298 (+ 27us) write_op.cc:260] PREPARE: finished
0810 23:32:59.921654 (+ 356us) write_op.cc:270] Start()
0810 23:32:59.921838 (+ 184us) write_op.cc:276] Timestamp: P: 1786404779921424 usec, L: 0
0810 23:32:59.921872 (+ 34us) op_driver.cc:331] REPLICATION: starting
0810 23:32:59.932088 (+ 10216us) log.cc:842] Serialized 433 byte log entry
0810 23:32:59.934132 (+ 2044us) spinlock_profiling.cc:243] Waited 9.31 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:33:00.256921 (+322789us) op_driver.cc:444] REPLICATION: finished
0810 23:33:00.259540 (+ 2619us) write_op.cc:301] APPLY: starting
0810 23:33:00.259637 (+ 97us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:00.259790 (+ 153us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:00.259809 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:00.260339 (+ 530us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:00.260511 (+ 172us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:33:00.260552 (+ 41us) write_op.cc:312] APPLY: finished
0810 23:33:00.260839 (+ 287us) log.cc:842] Serialized 29 byte log entry
0810 23:33:00.261029 (+ 190us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:00.261124 (+ 95us) write_op.cc:454] Released schema lock
0810 23:33:00.261176 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:00.261439 (+ 263us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:00.261485 (+ 46us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":422989952,"child_traces":[["op",{"apply.queue_time_us":2371,"apply.run_cpu_time_us":2213,"apply.run_wall_time_us":2214,"num_ops":2,"prepare.queue_time_us":3778,"prepare.run_cpu_time_us":4537,"prepare.run_wall_time_us":24464,"raft.queue_time_us":2103,"raft.run_cpu_time_us":19,"raft.run_wall_time_us":20,"replication_time_us":334833,"spinlock_wait_cycles":21424384}]]}
I20260810 23:33:00.274937 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_714aaba90fb0449bb9a5cb6ac00bae8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:00.197202 13514 tablet_bootstrap.cc:654] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:00.255540 12240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_3f5d5fb567214af3bcd7557741d52620" } modify_external_catalogs: true
I20260810 23:33:00.277062 12240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_3f5d5fb567214af3bcd7557741d52620" } modify_external_catalogs: true
I20260810 23:33:00.286414 12805 raft_consensus.cc:2454] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:33:00.288709 12224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 422b25457d1549e186641fcedd8feab9
I20260810 23:33:00.289366 13213 leader_election.cc:302] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:00.290584 12224 catalog_manager.cc:6565] ReplaceTablet: tablet 422b25457d1549e186641fcedd8feab9 deleted and replaced by tablet 9f3f4247a2d44088a02fbf82ac4e7c85
I20260810 23:33:00.290902 15217 raft_consensus.cc:2781] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:00.290979 12224 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 334) took 1225 ms. Trace:
I20260810 23:33:00.291409 15217 raft_consensus.cc:686] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:33:00.292308 15217 consensus_queue.cc:237] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:00.297298 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet 422b25457d1549e186641fcedd8feab9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f3f4247a2d44088a02fbf82ac4e7c85) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:00.301162 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 422b25457d1549e186641fcedd8feab9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f3f4247a2d44088a02fbf82ac4e7c85) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:00.301584 12229 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 333) took 1304 ms. Trace:
I20260810 23:33:00.301964 12229 rpcz_store.cc:276] 0810 23:32:58.996758 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.777615 (+780857us) service_pool.cc:227] Handling call
0810 23:33:00.009269 (+231654us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:33:00.011279 (+ 2010us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:33:00.300759 (+289480us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:33:00.301150 (+ 391us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:33:00.301496 (+ 346us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:00.017005 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:00.017566 (+ 561us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:00.017600 (+ 34us) write_op.cc:435] Acquired schema lock
0810 23:33:00.017615 (+ 15us) tablet.cc:660] Decoding operations
0810 23:33:00.018013 (+ 398us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:00.018062 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:00.018088 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:00.018431 (+ 343us) tablet.cc:699] Row locks acquired
0810 23:33:00.018443 (+ 12us) write_op.cc:260] PREPARE: finished
0810 23:33:00.018627 (+ 184us) write_op.cc:270] Start()
0810 23:33:00.018742 (+ 115us) write_op.cc:276] Timestamp: P: 1786404780018559 usec, L: 0
0810 23:33:00.018765 (+ 23us) op_driver.cc:331] REPLICATION: starting
0810 23:33:00.250501 (+231736us) spinlock_profiling.cc:243] Waited 68.8 ms on lock 0x7b5400032198. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce803f0c4 00007f0ce72eb4f9 00007f0ce72a6f91 00007f0ce72a6c6a 00007f0ce7312cd5 00007f0ce7313179 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9
0810 23:33:00.250899 (+ 398us) log.cc:842] Serialized 433 byte log entry
0810 23:33:00.283132 (+ 32233us) op_driver.cc:444] REPLICATION: finished
0810 23:33:00.286312 (+ 3180us) write_op.cc:301] APPLY: starting
0810 23:33:00.286371 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:00.286507 (+ 136us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:00.286524 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:00.286953 (+ 429us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:00.287107 (+ 154us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:33:00.287146 (+ 39us) write_op.cc:312] APPLY: finished
0810 23:33:00.287366 (+ 220us) log.cc:842] Serialized 29 byte log entry
0810 23:33:00.287655 (+ 289us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:00.287744 (+ 89us) write_op.cc:454] Released schema lock
0810 23:33:00.287778 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3037,"apply.run_cpu_time_us":2093,"apply.run_wall_time_us":18863,"num_ops":2,"prepare.queue_time_us":151,"prepare.run_cpu_time_us":6537,"prepare.run_wall_time_us":234521,"replication_time_us":264288,"spinlock_wait_cycles":158173440}]]}
I20260810 23:33:00.309126 12229 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_34a45145441d42cbbaf01a336059d4ec" } new_table_name: "default.Table_4a7751c5b61c4332adeeb9ee00474b27" modify_external_catalogs: true
I20260810 23:33:00.314584 13665 ts_tablet_manager.cc:1428] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.082s user 0.007s sys 0.000s
W20260810 23:33:00.336763 12952 consensus_peers.cc:487] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:00.337399 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 100231us
I20260810 23:33:00.340685 13022 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 271) took 1333 ms. Trace:
I20260810 23:33:00.341444 13665 tablet_bootstrap.cc:492] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:00.341130 13022 rpcz_store.cc:276] 0810 23:32:59.007453 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.007624 (+ 171us) service_pool.cc:227] Handling call
0810 23:32:59.123143 (+115519us) spinlock_profiling.cc:243] Waited 6.81 ms on lock 0x7b5800001898. stack: 00007fa86ed56033 00007fa86ed55f30 00007fa86e8d7534 00000000004db2b4 00000000004db249 00007fa86ebccf11 00007fa86ed50851 00007fa86ed50702 00007fa877be7b09 00007fa876c44730 00007fa876d26d21 00007fa876d041e3 00007fa876cb5869 00007fa8753a7c44 00007fa8753a7bd2 00007fa8753a7b01
0810 23:32:59.123301 (+ 158us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:59.123429 (+ 128us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:33:00.055128 (+931699us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:00.149845 (+ 94717us) tablet_replica.cc:205] Creating consensus instance
0810 23:33:00.340301 (+190456us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15674112}
I20260810 23:33:00.343228 13514 tablet_bootstrap.cc:492] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:33:00.343778 13514 ts_tablet_manager.cc:1397] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.180s user 0.012s sys 0.001s
I20260810 23:33:00.348453 13023 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 270) took 1319 ms. Trace:
I20260810 23:33:00.349622 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet 422b25457d1549e186641fcedd8feab9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f3f4247a2d44088a02fbf82ac4e7c85) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:00.291280 12224 rpcz_store.cc:276] 0810 23:32:59.065223 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.784160 (+718937us) service_pool.cc:227] Handling call
0810 23:33:00.290902 (+506742us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:32:59.953101 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:32:59.953678 (+ 577us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:32:59.953710 (+ 32us) write_op.cc:435] Acquired schema lock
0810 23:32:59.953729 (+ 19us) tablet.cc:660] Decoding operations
0810 23:32:59.954174 (+ 445us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:32:59.954219 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0810 23:32:59.954250 (+ 31us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:32:59.954504 (+ 254us) tablet.cc:699] Row locks acquired
0810 23:32:59.954523 (+ 19us) write_op.cc:260] PREPARE: finished
0810 23:32:59.954680 (+ 157us) write_op.cc:270] Start()
0810 23:32:59.954796 (+ 116us) write_op.cc:276] Timestamp: P: 1786404779954625 usec, L: 0
0810 23:32:59.954817 (+ 21us) op_driver.cc:331] REPLICATION: starting
0810 23:32:59.995689 (+ 40872us) log.cc:842] Serialized 631 byte log entry
0810 23:32:59.998142 (+ 2453us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:33:00.282770 (+284628us) op_driver.cc:444] REPLICATION: finished
0810 23:33:00.283471 (+ 701us) write_op.cc:301] APPLY: starting
0810 23:33:00.283538 (+ 67us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:00.283666 (+ 128us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:00.283688 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:00.284494 (+ 806us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:00.284760 (+ 266us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:00.284811 (+ 51us) write_op.cc:312] APPLY: finished
0810 23:33:00.285221 (+ 410us) log.cc:842] Serialized 29 byte log entry
0810 23:33:00.285431 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:00.285629 (+ 198us) write_op.cc:454] Released schema lock
0810 23:33:00.285671 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:00.285912 (+ 241us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:00.285938 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":504,"apply.run_cpu_time_us":2695,"apply.run_wall_time_us":2693,"num_ops":2,"prepare.queue_time_us":158,"prepare.run_cpu_time_us":5511,"prepare.run_wall_time_us":45475,"raft.queue_time_us":118,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":14,"replication_time_us":327857,"spinlock_wait_cycles":91686144}]]}
I20260810 23:33:00.349450 13023 rpcz_store.cc:276] 0810 23:32:59.006362 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.006563 (+ 201us) service_pool.cc:227] Handling call
0810 23:32:59.117829 (+111266us) spinlock_profiling.cc:243] Waited 6.75 ms on lock 0x7b5800001898. stack: 00007fa86ed56033 00007fa86ed55f30 00007fa86e8d7534 00000000004db2b4 00000000004db249 00007fa86ebccf11 00007fa86ed50851 00007fa86ed50702 00007fa877be7b09 00007fa876c44730 00007fa876d26d21 00007fa876d041e3 00007fa876cb5869 00007fa8753a7c44 00007fa8753a7bd2 00007fa8753a7b01
0810 23:32:59.118004 (+ 175us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:59.118130 (+ 126us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:59.340968 (+222838us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:00.162980 (+822012us) tablet_replica.cc:205] Creating consensus instance
0810 23:33:00.241398 (+ 78418us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14370,"spinlock_wait_cycles":15515008}
W20260810 23:33:00.366647 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_714aaba90fb0449bb9a5cb6ac00bae8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:00.368716 13665 tablet_bootstrap.cc:654] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:00.370155 13514 raft_consensus.cc:348] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:00.371016 13514 raft_consensus.cc:374] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:00.371295 13514 raft_consensus.cc:729] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:33:00.394650 13691 tablet_bootstrap.cc:654] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:00.402076 15250 ts_tablet_manager.cc:1918] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:00.402583 15250 log.cc:1196] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/10033af79e854103bc47a046d2219ca0
I20260810 23:33:00.415191 15250 ts_tablet_manager.cc:1939] T 10033af79e854103bc47a046d2219ca0 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:00.416846 13017 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58598 (request call id 272) took 1251 ms. Trace:
I20260810 23:33:00.422494 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 10033af79e854103bc47a046d2219ca0 (table default.Table_620c572fc75b44fe8ca892789875c1f7 [id=cf58351bb8594bdbb92168f093ec91a0]) successfully deleted
I20260810 23:33:00.423871 13514 consensus_queue.cc:260] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:00.424960 12236 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 335) took 1250 ms. Trace:
I20260810 23:33:00.417456 13017 rpcz_store.cc:276] 0810 23:32:59.165149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.165302 (+ 153us) service_pool.cc:227] Handling call
0810 23:32:59.166842 (+ 1540us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:59.166935 (+ 93us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:33:00.229908 (+1062973us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:00.396183 (+166275us) tablet_replica.cc:205] Creating consensus instance
0810 23:33:00.416550 (+ 20367us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:33:00.425302 12236 rpcz_store.cc:276] 0810 23:32:59.174217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.815722 (+641505us) service_pool.cc:227] Handling call
0810 23:32:59.993433 (+177711us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7156 00007f0ce6e24034 00007f0ce6e23fc2 00007f0ce6e23ef1 00007f0ce6e23e6c 00007f0ce6e23122 00007f0ce50c694c 00007f0ce50c5d86
0810 23:33:00.013653 (+ 20220us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:33:00.015239 (+ 1586us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:33:00.424188 (+408949us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:33:00.424582 (+ 394us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:33:00.424884 (+ 302us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:00.258372 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:00.259002 (+ 630us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:00.259029 (+ 27us) write_op.cc:435] Acquired schema lock
0810 23:33:00.259045 (+ 16us) tablet.cc:660] Decoding operations
0810 23:33:00.259578 (+ 533us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:00.259626 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:00.259655 (+ 29us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:00.259889 (+ 234us) tablet.cc:699] Row locks acquired
0810 23:33:00.259902 (+ 13us) write_op.cc:260] PREPARE: finished
0810 23:33:00.260078 (+ 176us) write_op.cc:270] Start()
0810 23:33:00.260303 (+ 225us) write_op.cc:276] Timestamp: P: 1786404780259972 usec, L: 0
0810 23:33:00.260322 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 23:33:00.261118 (+ 796us) log.cc:842] Serialized 433 byte log entry
0810 23:33:00.404978 (+143860us) op_driver.cc:444] REPLICATION: finished
0810 23:33:00.407242 (+ 2264us) write_op.cc:301] APPLY: starting
0810 23:33:00.407312 (+ 70us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:00.407458 (+ 146us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:00.407624 (+ 166us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:00.421771 (+ 14147us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:00.421963 (+ 192us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:33:00.422013 (+ 50us) write_op.cc:312] APPLY: finished
0810 23:33:00.423450 (+ 1437us) log.cc:842] Serialized 29 byte log entry
0810 23:33:00.423679 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:00.423778 (+ 99us) write_op.cc:454] Released schema lock
0810 23:33:00.423813 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:00.424066 (+ 253us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:00.424089 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":51505024,"child_traces":[["op",{"apply.queue_time_us":2105,"apply.run_cpu_time_us":2407,"apply.run_wall_time_us":17029,"num_ops":2,"prepare.queue_time_us":178057,"prepare.run_cpu_time_us":2644,"prepare.run_wall_time_us":3011,"replication_time_us":144558}]]}
I20260810 23:33:00.446089 14121 raft_consensus.cc:2229] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:00.455742 15216 raft_consensus.cc:482] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:00.456542 15216 raft_consensus.cc:504] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:33:00.461588 12229 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff7bd9b6ca4845d8b8705ff970c8f06e consensus state has no leader
I20260810 23:33:00.462543 15216 leader_election.cc:288] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:00.463542 15348 consensus_queue.cc:1035] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260810 23:33:00.474516 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e8311c941b43058384cee6d8263dc1" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:00.475522 13060 raft_consensus.cc:2454] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:00.491743 14121 raft_consensus.cc:2258] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:00.492333 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e8311c941b43058384cee6d8263dc1" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:00.493310 13058 raft_consensus.cc:3037] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:00.509645 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9443fb32d79941d58b13607e8ac064cd" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:00.510474 13062 raft_consensus.cc:3037] T 9443fb32d79941d58b13607e8ac064cd P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:00.512727 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9443fb32d79941d58b13607e8ac064cd" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:00.513868 13514 ts_tablet_manager.cc:1428] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.089s user 0.007s sys 0.000s
I20260810 23:33:00.516052 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_94d5b6058b6041b79fb4ac9e4e07f354"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:00.516829 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb2ee1c9b41478da001aa67d7a7ad3c" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:33:00.518270 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94d5b6058b6041b79fb4ac9e4e07f354 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:00.507086 12229 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_be4c5de03d7649a08296569cbe6a3503"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:00.529893 13514 tablet_bootstrap.cc:492] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:33:00.537555 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 80069.5us
I20260810 23:33:00.540571 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7bd9b6ca4845d8b8705ff970c8f06e" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
W20260810 23:33:00.540973 12229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be4c5de03d7649a08296569cbe6a3503 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:00.541491 12806 raft_consensus.cc:2454] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:00.544574 13213 leader_election.cc:302] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:00.545359 15216 consensus_queue.cc:1035] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:33:00.545874 15156 raft_consensus.cc:2781] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:00.553913 15156 raft_consensus.cc:482] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 23:33:00.555330 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 92b81e7056a441d2854ba5ab6b4c5433 because the tablet was not found. No further retry: Not found: Tablet not found: 92b81e7056a441d2854ba5ab6b4c5433
I20260810 23:33:00.554811 15156 raft_consensus.cc:3037] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:33:00.565937 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet c3d4c879ca6a402dbafe91940e7b367a because the tablet was not found. No further retry: Not found: Tablet not found: c3d4c879ca6a402dbafe91940e7b367a
W20260810 23:33:00.577144 15350 log.cc:924] Time spent T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Append to log took a long time: real 0.058s user 0.002s sys 0.003s
W20260810 23:33:00.570897 15361 log.cc:924] Time spent T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3: Append to log took a long time: real 0.070s user 0.002s sys 0.000s
I20260810 23:33:00.568471 15250 tablet_replica.cc:330] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:00.591894 13665 tablet_bootstrap.cc:492] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:00.613421 14121 ts_tablet_manager.cc:1905] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:00.627350 13514 tablet_bootstrap.cc:654] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:00.523970 13062 raft_consensus.cc:2454] T 9443fb32d79941d58b13607e8ac064cd P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
W20260810 23:33:00.628777 12952 consensus_peers.cc:487] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:00.579947 15356 log.cc:924] Time spent T cd24471dd52149d395808493ad9fa68c P f54aa78a60784a3bb2f5eb4bb2401ce3: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260810 23:33:00.640432 13665 ts_tablet_manager.cc:1397] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.310s user 0.015s sys 0.010s
W20260810 23:33:00.551862 15374 log.cc:924] Time spent T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20260810 23:33:00.618830 15156 raft_consensus.cc:504] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.644529 15156 leader_election.cc:288] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:00.645305 13665 raft_consensus.cc:348] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.646024 13665 raft_consensus.cc:374] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:00.646265 13665 raft_consensus.cc:729] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:00.646107 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7bd9b6ca4845d8b8705ff970c8f06e" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:00.646814 12805 raft_consensus.cc:3037] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:00.648007 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_b78ae79905d8448085a6e318220c1584"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:00.657292 15250 raft_consensus.cc:2229] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:00.657593 12805 raft_consensus.cc:2454] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:33:00.658469 15250 raft_consensus.cc:2258] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:00.659080 13213 leader_election.cc:302] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:00.660027 15156 raft_consensus.cc:2781] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:00.660616 15156 raft_consensus.cc:686] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
W20260810 23:33:00.662165 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b78ae79905d8448085a6e318220c1584 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:00.662199 15156 consensus_queue.cc:237] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.675478 15198 raft_consensus.cc:482] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:00.678766 12236 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_ac23e0995fcd414ead1b88406cf7d79e" } new_table_name: "default.Table_970b3cb3161f4f5a9fec0f3dfa72f1a5" modify_external_catalogs: true
I20260810 23:33:00.614861 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb2ee1c9b41478da001aa67d7a7ad3c" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
W20260810 23:33:00.680778 12698 consensus_peers.cc:487] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f935119e4d9b4accb79db98185c80aca. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:00.676409 15198 raft_consensus.cc:504] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:00.686159 15198 leader_election.cc:288] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:00.686417 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c39d751e544d9a8afa26dc935a819a" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:33:00.689867 12698 leader_election.cc:341] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:00.703243 15383 tablet.cc:1722] T c01ede4b52c3498a924340b39c3c2435 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:33:00.724654 15250 ts_tablet_manager.cc:1905] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:00.722615 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7bd9b6ca4845d8b8705ff970c8f06e" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:00.725754 13058 raft_consensus.cc:2454] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:33:00.734011 13691 tablet_bootstrap.cc:492] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:00.734618 13691 ts_tablet_manager.cc:1397] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.493s user 0.019s sys 0.005s
I20260810 23:33:00.736752 13665 consensus_queue.cc:260] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.780783 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7bd9b6ca4845d8b8705ff970c8f06e" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:00.775831 12958 leader_election.cc:302] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:00.785351 15321 raft_consensus.cc:2726] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:00.786798 15373 raft_consensus.cc:482] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:00.787533 15373 raft_consensus.cc:504] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.791015 15373 leader_election.cc:288] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:00.793835 13665 ts_tablet_manager.cc:1428] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.153s user 0.007s sys 0.000s
I20260810 23:33:00.803207 15391 tablet.cc:1722] T c01ede4b52c3498a924340b39c3c2435 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:33:00.813237 13665 tablet_bootstrap.cc:492] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:00.826604 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_90b34e5db74b40c1954de96c5c9fa3d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:00.826998 12789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240adfa4be844518bbc9624d0e408789" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:00.828022 12789 raft_consensus.cc:2454] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
W20260810 23:33:00.830338 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90b34e5db74b40c1954de96c5c9fa3d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:00.836306 13212 consensus_peers.cc:487] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Couldn't send request to peer f26517150f1c480a877f15aec71d020b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:00.838567 13212 consensus_peers.cc:487] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Couldn't send request to peer f26517150f1c480a877f15aec71d020b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:00.842271 13665 tablet_bootstrap.cc:654] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:00.847348 13691 raft_consensus.cc:348] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.849399 13691 raft_consensus.cc:374] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:00.849787 13691 raft_consensus.cc:729] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
W20260810 23:33:00.851858 13212 leader_election.cc:341] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:00.854547 13212 leader_election.cc:341] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:00.868265 15250 ts_tablet_manager.cc:1918] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260810 23:33:00.869741 13212 leader_election.cc:341] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:00.874374 13212 leader_election.cc:341] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:00.875762 15217 raft_consensus.cc:482] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:00.876245 15217 raft_consensus.cc:504] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.879195 15217 leader_election.cc:288] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:00.881554 15156 consensus_queue.cc:1035] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:33:00.883024 15250 log.cc:1196] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/4cdec13ee70d402fa3a08119e7bf1313
I20260810 23:33:00.883935 15250 ts_tablet_manager.cc:1939] T 4cdec13ee70d402fa3a08119e7bf1313 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:00.862519 15082 ts_tablet_manager.cc:1918] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:00.887456 15082 log.cc:1196] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/4f3049bcdc6e43c795961c74f0d008f5
I20260810 23:33:00.888562 15082 ts_tablet_manager.cc:1939] T 4f3049bcdc6e43c795961c74f0d008f5 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:33:00.889967 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 275) took 3222 ms. Trace:
I20260810 23:33:00.890395 15082 rpcz_store.cc:276] 0810 23:32:57.668086 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.668331 (+ 245us) service_pool.cc:227] Handling call
0810 23:33:00.199235 (+2530904us) ts_tablet_manager.cc:1145] Deleting tablet 4f3049bcdc6e43c795961c74f0d008f5
0810 23:33:00.199315 (+ 80us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:00.374449 (+175134us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:00.829656 (+455207us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:00.889876 (+ 60220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2529677}
I20260810 23:33:00.891918 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 4f3049bcdc6e43c795961c74f0d008f5 (table default.Table_7548a45d866747b9b0944e13b211c643 [id=c6af3828fca045bdbe16438171ff83cd]) successfully deleted
I20260810 23:33:00.892268 15082 tablet_replica.cc:330] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:33:00.900019 15250 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 283) took 1185 ms. Trace:
I20260810 23:33:00.900833 15250 rpcz_store.cc:276] 0810 23:32:59.714790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.714960 (+ 170us) service_pool.cc:227] Handling call
0810 23:33:00.567674 (+852714us) ts_tablet_manager.cc:1145] Deleting tablet 4cdec13ee70d402fa3a08119e7bf1313
0810 23:33:00.567772 (+ 98us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:00.836026 (+268254us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:00.867935 (+ 31909us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:00.896247 (+ 28312us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b5800001898. stack: 00007f8764d74033 00007f8764d73f30 00007f87648f5534 00000000004db2b4 00000000004db249 00007f8764beaf11 00007f8764d6e851 00007f8764d6ee82 00007f876dc2e8b9 00007f876cc625d8 00007f876cd26e42 00007f876cd56e10 00007f876cd3de0d 00007f876cd3ddb9 00007f876cd3dd49 00007f876cd3dd11
0810 23:33:00.899889 (+ 3642us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24231808,"tablet-delete.queue_time_us":848480}
I20260810 23:33:00.903375 15250 tablet_replica.cc:330] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:00.903645 13514 tablet_bootstrap.cc:492] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:33:00.904397 13514 ts_tablet_manager.cc:1397] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.375s user 0.014s sys 0.005s
I20260810 23:33:00.906075 15250 raft_consensus.cc:2229] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:00.909179 15250 raft_consensus.cc:2258] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:33:00.914214 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 115976us
W20260810 23:33:00.915159 12700 leader_election.cc:341] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:00.919405 15250 ts_tablet_manager.cc:1905] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:00.921221 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 4cdec13ee70d402fa3a08119e7bf1313 (table default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]) successfully deleted
I20260810 23:33:00.923967 13062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c39d751e544d9a8afa26dc935a819a" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:00.930850 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a480093c481941f0bb35949c7606f37f" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:00.925721 13691 consensus_queue.cc:260] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.937391 12700 leader_election.cc:302] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:00.921367 15082 raft_consensus.cc:2229] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:00.917531 13514 raft_consensus.cc:348] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:00.940713 13514 raft_consensus.cc:374] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:00.941258 13514 raft_consensus.cc:729] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:33:00.942312 13514 consensus_queue.cc:260] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:00.955766 15250 ts_tablet_manager.cc:1918] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:00.956259 15250 log.cc:1196] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/f47b8b57d5b44a018bc2d7b3f976d803
I20260810 23:33:00.956458 15198 raft_consensus.cc:2726] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:00.957115 15250 ts_tablet_manager.cc:1939] T f47b8b57d5b44a018bc2d7b3f976d803 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:00.959951 15250 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 285) took 1185 ms. Trace:
I20260810 23:33:00.960327 15250 rpcz_store.cc:276] 0810 23:32:59.774897 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.775060 (+ 163us) service_pool.cc:227] Handling call
0810 23:33:00.903053 (+1127993us) ts_tablet_manager.cc:1145] Deleting tablet f47b8b57d5b44a018bc2d7b3f976d803
0810 23:33:00.903109 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:00.945309 (+ 42200us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:00.955439 (+ 10130us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:00.959881 (+ 4442us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1127062}
I20260810 23:33:00.962035 15250 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 284) took 1188 ms. Trace:
I20260810 23:33:00.962350 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet f47b8b57d5b44a018bc2d7b3f976d803 (table default.Table_7ba7fd44e98542e78c604739a5086a91 [id=a7a55f9e68f04a9c9c43b9607cd901c6]) successfully deleted
I20260810 23:33:00.962358 15250 rpcz_store.cc:276] 0810 23:32:59.773661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.780127 (+ 6466us) service_pool.cc:227] Handling call
0810 23:33:00.961695 (+1181568us) ts_tablet_manager.cc:1145] Deleting tablet 423e8d583d704fa9834026ebcd57e17e
0810 23:33:00.961762 (+ 67us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:00.961981 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1180736}
I20260810 23:33:00.962836 15082 raft_consensus.cc:2258] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:33:00.963989 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 423e8d583d704fa9834026ebcd57e17e because the tablet was not found. No further retry: Not found: Tablet not found: 423e8d583d704fa9834026ebcd57e17e
I20260810 23:33:00.965112 15250 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 286) took 1158 ms. Trace:
I20260810 23:33:00.965427 15250 rpcz_store.cc:276] 0810 23:32:59.806995 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.809968 (+ 2973us) service_pool.cc:227] Handling call
0810 23:33:00.964819 (+1154851us) ts_tablet_manager.cc:1145] Deleting tablet 4d697e7533af4fc0a15cb41f3fcce4db
0810 23:33:00.964864 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:00.965062 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1089530}
W20260810 23:33:00.966858 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 4d697e7533af4fc0a15cb41f3fcce4db because the tablet was not found. No further retry: Not found: Tablet not found: 4d697e7533af4fc0a15cb41f3fcce4db
I20260810 23:33:00.967041 15082 ts_tablet_manager.cc:1905] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:00.968297 15250 tablet_replica.cc:330] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:00.968703 15373 raft_consensus.cc:482] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:33:00.969372 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 54262.5us
I20260810 23:33:00.969295 15373 raft_consensus.cc:504] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:00.969725 15250 raft_consensus.cc:2229] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 23:33:00.970257 12700 leader_election.cc:341] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:00.970548 15250 raft_consensus.cc:2258] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:00.970706 12700 leader_election.cc:302] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:33:00.971591 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 98471.8us
W20260810 23:33:00.972658 12958 consensus_peers.cc:487] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:00.978340 15373 leader_election.cc:288] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:00.983070 15250 ts_tablet_manager.cc:1905] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:00.983830 13017 tablet_service.cc:1467] Processing CreateTablet for tablet bb265c1715ca4f1dad33c6ca36ce019d (DEFAULT_TABLE table=default.Table_afb094287f0e4e88aefdd128d2fd7ef8 [id=39369f2619cf417ea917db1973d0270b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:00.985349 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb265c1715ca4f1dad33c6ca36ce019d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:00.989180 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet f47b8b57d5b44a018bc2d7b3f976d803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:47546
W20260810 23:33:00.989230 12228 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 65) took 1375 ms (client timeout 1000 ms). Trace:
I20260810 23:33:00.990934 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet a480093c481941f0bb35949c7606f37f with delete_type TABLET_DATA_DELETED (replaced by tablet 0703cf71490c4c71a95f55863583a03c) from {username='slave'} at 127.0.0.1:47546
W20260810 23:33:00.991716 12228 rpcz_store.cc:269] 0810 23:32:59.614091 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.638669 (+ 24578us) service_pool.cc:227] Handling call
0810 23:32:59.687729 (+ 49060us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:33:00.989139 (+1301410us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":83427584}
I20260810 23:33:00.994645 12229 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_d028b6b163de469a9325b241cb5701a1 [id=49a3a44dd7f9473d9c025d8690a00c1f]). Expected version 3 got 2
I20260810 23:33:00.996040 13291 tablet_service.cc:1467] Processing CreateTablet for tablet bb265c1715ca4f1dad33c6ca36ce019d (DEFAULT_TABLE table=default.Table_afb094287f0e4e88aefdd128d2fd7ef8 [id=39369f2619cf417ea917db1973d0270b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:00.997236 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb265c1715ca4f1dad33c6ca36ce019d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:00.998348 12229 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet c01ede4b52c3498a924340b39c3c2435 (table default.Table_ebe848b2f9ba4077a556369c87aa436a [id=ae7b5faec4c34d35981e0a1d464202ef]). Expected version 1 got 0
I20260810 23:33:00.999130 12229 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet e6b8bf38da5647889756935404caad75 (table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]). Expected version 1 got 0
I20260810 23:33:01.001679 13665 tablet_bootstrap.cc:492] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:01.002156 13665 ts_tablet_manager.cc:1397] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.201s user 0.029s sys 0.017s
I20260810 23:33:01.003748 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_a8f723dad5a54fceb62622c2aefe987c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:01.005534 13665 raft_consensus.cc:348] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:33:01.005861 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8f723dad5a54fceb62622c2aefe987c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:01.009279 12755 tablet_service.cc:1467] Processing CreateTablet for tablet bb265c1715ca4f1dad33c6ca36ce019d (DEFAULT_TABLE table=default.Table_afb094287f0e4e88aefdd128d2fd7ef8 [id=39369f2619cf417ea917db1973d0270b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.000803 12229 catalog_manager.cc:5495] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b reported cstate change: term changed from 0 to 1, leader changed from <none> to f26517150f1c480a877f15aec71d020b (127.11.226.130). New cstate: current_term: 1 leader_uuid: "f26517150f1c480a877f15aec71d020b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } }
I20260810 23:33:01.018329 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_1de5219e712b481d8048407bd4cec52a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:01.020620 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1de5219e712b481d8048407bd4cec52a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:01.016542 15250 ts_tablet_manager.cc:1918] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:01.028438 13665 raft_consensus.cc:374] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:01.028800 13665 raft_consensus.cc:729] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:01.029896 13665 consensus_queue.cc:260] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:01.033329 13274 tablet_service.cc:1467] Processing CreateTablet for tablet be39e8eb12764ba386c62383e1de659b (DEFAULT_TABLE table=default.Table_a3a059d2601345d68eadf189772bcde5 [id=5cc3da1483184129834caaad3e509c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.035202 13274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be39e8eb12764ba386c62383e1de659b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.039124 15250 log.cc:1196] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/4f3049bcdc6e43c795961c74f0d008f5
I20260810 23:33:01.040644 15250 ts_tablet_manager.cc:1939] T 4f3049bcdc6e43c795961c74f0d008f5 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:01.045532 15410 raft_consensus.cc:2726] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:01.047381 13514 ts_tablet_manager.cc:1428] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.143s user 0.004s sys 0.004s
I20260810 23:33:01.052166 12234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd
I20260810 23:33:01.054841 12234 catalog_manager.cc:6565] ReplaceTablet: tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd deleted and replaced by tablet 46450c84a9054fbdbb5523ec835d43d2
I20260810 23:33:01.054924 13273 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd with delete_type TABLET_DATA_DELETED (replaced by tablet 46450c84a9054fbdbb5523ec835d43d2) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:01.055341 12234 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:34742 (request call id 336) took 1874 ms. Trace:
I20260810 23:33:01.055469 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd with delete_type TABLET_DATA_DELETED (replaced by tablet 46450c84a9054fbdbb5523ec835d43d2) from {username='slave'} at 127.0.0.1:47546
W20260810 23:33:01.048620 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 78409.6us
W20260810 23:33:01.058362 12700 leader_election.cc:341] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:01.055707 12234 rpcz_store.cc:276] 0810 23:32:59.181207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.962193 (+780986us) service_pool.cc:227] Handling call
0810 23:33:01.055252 (+1093059us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:00.993656 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:00.994856 (+ 1200us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:00.994885 (+ 29us) write_op.cc:435] Acquired schema lock
0810 23:33:00.994903 (+ 18us) tablet.cc:660] Decoding operations
0810 23:33:00.995295 (+ 392us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:00.995350 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:00.995380 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:00.995629 (+ 249us) tablet.cc:699] Row locks acquired
0810 23:33:00.995645 (+ 16us) write_op.cc:260] PREPARE: finished
0810 23:33:00.995777 (+ 132us) write_op.cc:270] Start()
0810 23:33:00.995873 (+ 96us) write_op.cc:276] Timestamp: P: 1786404780995688 usec, L: 0
0810 23:33:00.995893 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:33:01.002451 (+ 6558us) log.cc:842] Serialized 631 byte log entry
0810 23:33:01.026711 (+ 24260us) spinlock_profiling.cc:243] Waited 5.71 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:33:01.041146 (+ 14435us) op_driver.cc:444] REPLICATION: finished
0810 23:33:01.049417 (+ 8271us) write_op.cc:301] APPLY: starting
0810 23:33:01.049484 (+ 67us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:01.049620 (+ 136us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:01.049641 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:01.050040 (+ 399us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:01.050273 (+ 233us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:01.050331 (+ 58us) write_op.cc:312] APPLY: finished
0810 23:33:01.050552 (+ 221us) log.cc:842] Serialized 29 byte log entry
0810 23:33:01.051275 (+ 723us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:01.051369 (+ 94us) write_op.cc:454] Released schema lock
0810 23:33:01.051406 (+ 37us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":7868,"num_ops":2,"prepare.queue_time_us":9819,"prepare.run_cpu_time_us":4845,"prepare.run_wall_time_us":35626,"raft.queue_time_us":318,"raft.run_cpu_time_us":1451,"raft.run_wall_time_us":1451,"replication_time_us":45170,"spinlock_wait_cycles":13140736}]]}
I20260810 23:33:01.058840 12700 leader_election.cc:302] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:01.064402 15427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.189:42101 every 8 connection(s)
I20260810 23:33:01.065150 12764 tablet_service.cc:1467] Processing CreateTablet for tablet be39e8eb12764ba386c62383e1de659b (DEFAULT_TABLE table=default.Table_a3a059d2601345d68eadf189772bcde5 [id=5cc3da1483184129834caaad3e509c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.068404 13549 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.189:42101
I20260810 23:33:01.070554 15250 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 287) took 1042 ms. Trace:
I20260810 23:33:01.070926 15250 rpcz_store.cc:276] 0810 23:33:00.028432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.028617 (+ 185us) service_pool.cc:227] Handling call
0810 23:33:00.967943 (+939326us) ts_tablet_manager.cc:1145] Deleting tablet 4f3049bcdc6e43c795961c74f0d008f5
0810 23:33:00.968099 (+ 156us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:01.007300 (+ 39201us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:01.016231 (+ 8931us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:01.070468 (+ 54237us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19992,"tablet-delete.queue_time_us":938304}
I20260810 23:33:01.072737 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 4f3049bcdc6e43c795961c74f0d008f5 (table default.Table_7548a45d866747b9b0944e13b211c643 [id=c6af3828fca045bdbe16438171ff83cd]) successfully deleted
W20260810 23:33:01.075491 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (table default.Table_4a7751c5b61c4332adeeb9ee00474b27 [id=2c8320255aab4036b5350d082f9e04ef], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet ff7bd9b6ca4845d8b8705ff970c8f06e consensus state has no leader
W20260810 23:33:01.078217 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 422b25457d1549e186641fcedd8feab9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 422b25457d1549e186641fcedd8feab9 already in progress: creating tablet
I20260810 23:33:01.078800 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb265c1715ca4f1dad33c6ca36ce019d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:01.078989 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd already in progress: creating tablet
I20260810 23:33:01.085116 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be39e8eb12764ba386c62383e1de659b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.083675 13665 ts_tablet_manager.cc:1428] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.081s user 0.007s sys 0.001s
I20260810 23:33:01.087251 13273 tablet_service.cc:1467] Processing CreateTablet for tablet 7031c257fdf34a28b6954d3d4827a8da (DEFAULT_TABLE table=default.Table_f64ee2d577744a919f65f54c28ccc88b [id=ca7378a0859a467dbd7c50eff469e96d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.088719 13665 tablet_bootstrap.cc:492] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:01.093827 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 13549
I20260810 23:33:01.097709 13273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7031c257fdf34a28b6954d3d4827a8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.085090 13514 tablet_bootstrap.cc:492] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:33:01.105011 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 46678.7us
W20260810 23:33:01.106033 12700 leader_election.cc:341] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:01.106623 12700 leader_election.cc:302] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:01.107034 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_86a07768a7834d14812dbdd318a66e5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:01.109100 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86a07768a7834d14812dbdd318a66e5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:01.113164 15421 tablet.cc:1722] T e6b8bf38da5647889756935404caad75 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:33:01.088204 13549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/master-1/data/info.pb
I20260810 23:33:01.130409 12226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 32fad37a28e14903b8bde5aac6ee4e69
I20260810 23:33:01.131412 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 7031c257fdf34a28b6954d3d4827a8da (DEFAULT_TABLE table=default.Table_f64ee2d577744a919f65f54c28ccc88b [id=ca7378a0859a467dbd7c50eff469e96d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.132174 12226 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 339) took 1292 ms. Trace:
I20260810 23:33:01.132668 13277 tablet_service.cc:1514] Processing DeleteTablet for tablet 32fad37a28e14903b8bde5aac6ee4e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:00 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:01.133129 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7031c257fdf34a28b6954d3d4827a8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.132483 12226 rpcz_store.cc:276] 0810 23:32:59.839972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.254750 (+414778us) service_pool.cc:227] Handling call
0810 23:33:00.257756 (+ 3006us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:33:00.335585 (+ 77829us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:33:00.335817 (+ 232us) catalog_manager.cc:2761] Locking tablets
0810 23:33:00.984088 (+648271us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:33:01.129824 (+145736us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:33:01.130167 (+ 343us) catalog_manager.cc:2809] Committing in-memory state
0810 23:33:01.130299 (+ 132us) catalog_manager.cc:2813] Aborting table tasks
0810 23:33:01.132106 (+ 1807us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:01.029616 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:01.030256 (+ 640us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:01.030295 (+ 39us) write_op.cc:435] Acquired schema lock
0810 23:33:01.030319 (+ 24us) tablet.cc:660] Decoding operations
0810 23:33:01.030808 (+ 489us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:01.030872 (+ 64us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:01.030909 (+ 37us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:01.049490 (+ 18581us) tablet.cc:699] Row locks acquired
0810 23:33:01.049514 (+ 24us) write_op.cc:260] PREPARE: finished
0810 23:33:01.049694 (+ 180us) write_op.cc:270] Start()
0810 23:33:01.049826 (+ 132us) write_op.cc:276] Timestamp: P: 1786404781049588 usec, L: 0
0810 23:33:01.049855 (+ 29us) op_driver.cc:331] REPLICATION: starting
0810 23:33:01.103582 (+ 53727us) spinlock_profiling.cc:243] Waited 52.5 ms on lock 0x7b5400032120. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce72acdc0 00007f0ce731dbd2 00007f0ce73132d7 00007f0ce7313114 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1
0810 23:33:01.104394 (+ 812us) log.cc:842] Serialized 765 byte log entry
0810 23:33:01.125238 (+ 20844us) op_driver.cc:444] REPLICATION: finished
0810 23:33:01.127607 (+ 2369us) write_op.cc:301] APPLY: starting
0810 23:33:01.127710 (+ 103us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:01.127966 (+ 256us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:01.128015 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:01.128610 (+ 595us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:01.128755 (+ 145us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:33:01.128807 (+ 52us) write_op.cc:312] APPLY: finished
0810 23:33:01.129029 (+ 222us) log.cc:842] Serialized 29 byte log entry
0810 23:33:01.129230 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:01.129362 (+ 132us) write_op.cc:454] Released schema lock
0810 23:33:01.129427 (+ 65us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":1897856,"child_traces":[["op",{"apply.queue_time_us":2058,"num_ops":2,"prepare.queue_time_us":43092,"raft.queue_time_us":102,"raft.run_cpu_time_us":746,"raft.run_wall_time_us":746,"replication_time_us":75212,"spinlock_wait_cycles":120699264}]]}
W20260810 23:33:01.136233 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 32fad37a28e14903b8bde5aac6ee4e69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32fad37a28e14903b8bde5aac6ee4e69 already in progress: creating tablet
I20260810 23:33:01.172873 12763 tablet_service.cc:1514] Processing DeleteTablet for tablet 32fad37a28e14903b8bde5aac6ee4e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:00 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:01.173641 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_496ebacf57f14572b620d213d8865579"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:01.175519 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_496ebacf57f14572b620d213d8865579 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:01.186062 13273 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0e69195596486fb03256baa22446e0 (DEFAULT_TABLE table=default.Table_763854d208924ff78919903a71a7c826 [id=575025603ae44b0785de76cc5a28ab9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.188428 13273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0e69195596486fb03256baa22446e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.164816 15410 raft_consensus.cc:2726] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:01.190958 12763 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0e69195596486fb03256baa22446e0 (DEFAULT_TABLE table=default.Table_763854d208924ff78919903a71a7c826 [id=575025603ae44b0785de76cc5a28ab9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.196341 13665 tablet_bootstrap.cc:654] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:01.196543 12240 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 9f3f4247a2d44088a02fbf82ac4e7c85
I20260810 23:33:01.196995 12240 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 338) took 1403 ms. Trace:
I20260810 23:33:01.197290 12240 rpcz_store.cc:276] 0810 23:32:59.794070 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.020146 (+226076us) service_pool.cc:227] Handling call
0810 23:33:00.255360 (+235214us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7466 00007f0ce6e28b34 00007f0ce6e28ac2 00007f0ce6e289f1 00007f0ce6e2896c 00007f0ce6e27c22 00007f0ce50c694c 00007f0ce50c5d86
0810 23:33:00.277602 (+ 22242us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:33:00.278670 (+ 1068us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:33:00.278846 (+ 176us) catalog_manager.cc:2761] Locking tablets
0810 23:33:00.989358 (+710512us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:33:01.195925 (+206567us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:33:01.196276 (+ 351us) catalog_manager.cc:2809] Committing in-memory state
0810 23:33:01.196419 (+ 143us) catalog_manager.cc:2813] Aborting table tasks
0810 23:33:01.196931 (+ 512us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:01.157102 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:01.158198 (+ 1096us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:01.158259 (+ 61us) write_op.cc:435] Acquired schema lock
0810 23:33:01.158273 (+ 14us) tablet.cc:660] Decoding operations
0810 23:33:01.158990 (+ 717us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:01.159077 (+ 87us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:01.159133 (+ 56us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:01.172895 (+ 13762us) tablet.cc:699] Row locks acquired
0810 23:33:01.172912 (+ 17us) write_op.cc:260] PREPARE: finished
0810 23:33:01.173090 (+ 178us) write_op.cc:270] Start()
0810 23:33:01.173229 (+ 139us) write_op.cc:276] Timestamp: P: 1786404781172980 usec, L: 0
0810 23:33:01.173246 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:33:01.181797 (+ 8551us) log.cc:842] Serialized 734 byte log entry
0810 23:33:01.193731 (+ 11934us) op_driver.cc:444] REPLICATION: finished
0810 23:33:01.194319 (+ 588us) write_op.cc:301] APPLY: starting
0810 23:33:01.194413 (+ 94us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:01.194540 (+ 127us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:01.194640 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:01.194905 (+ 265us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:01.195094 (+ 189us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:33:01.195138 (+ 44us) write_op.cc:312] APPLY: finished
0810 23:33:01.195374 (+ 236us) log.cc:842] Serialized 29 byte log entry
0810 23:33:01.195545 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:01.195636 (+ 91us) write_op.cc:454] Released schema lock
0810 23:33:01.195669 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:01.195873 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:01.195898 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":536596736,"child_traces":[["op",{"apply.queue_time_us":411,"apply.run_cpu_time_us":1897,"apply.run_wall_time_us":2932,"num_ops":2,"prepare.queue_time_us":165462,"prepare.run_cpu_time_us":4622,"prepare.run_wall_time_us":25121,"raft.queue_time_us":980,"raft.run_cpu_time_us":1157,"raft.run_wall_time_us":1190,"replication_time_us":20390,"spinlock_wait_cycles":190720}]]}
W20260810 23:33:01.185604 12700 raft_consensus.cc:2670] T a41ddf9f6f71443ba75c97d2b010372c P 191d120f4c7f44fc8475305a6d2e75f3: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260810 23:33:01.227578 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 121568us
W20260810 23:33:01.229022 12700 leader_election.cc:341] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:01.229912 12700 leader_election.cc:302] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:01.236080 12228 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_d860efaf3e3a4d1386d4c8e328463a8a" } new_table_name: "default.Table_e2972dea79e44002b43aac315db921ca" modify_external_catalogs: true
W20260810 23:33:01.249079 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 163069us
I20260810 23:33:01.251127 13665 tablet_bootstrap.cc:492] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:01.251592 13665 ts_tablet_manager.cc:1397] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.163s user 0.008s sys 0.005s
I20260810 23:33:01.260892 13023 tablet_service.cc:1467] Processing CreateTablet for tablet be39e8eb12764ba386c62383e1de659b (DEFAULT_TABLE table=default.Table_a3a059d2601345d68eadf189772bcde5 [id=5cc3da1483184129834caaad3e509c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.265691 13291 tablet_service.cc:1467] Processing CreateTablet for tablet d3f280d8dbc048e5bcda6c5e9b2a7387 (DEFAULT_TABLE table=default.Table_d9fd2a26e54c483ebbc082b5a4ce014f [id=deb4795f344f4f108092842f7bd08934]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.267267 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f280d8dbc048e5bcda6c5e9b2a7387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.267697 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be39e8eb12764ba386c62383e1de659b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.298692 15130 consensus_queue.cc:1035] T cd24471dd52149d395808493ad9fa68c P f26517150f1c480a877f15aec71d020b [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260810 23:33:01.306608 13665 raft_consensus.cc:348] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:01.307529 13665 raft_consensus.cc:374] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:01.307818 13665 raft_consensus.cc:729] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:01.270602 12768 tablet_service.cc:1467] Processing CreateTablet for tablet d3f280d8dbc048e5bcda6c5e9b2a7387 (DEFAULT_TABLE table=default.Table_d9fd2a26e54c483ebbc082b5a4ce014f [id=deb4795f344f4f108092842f7bd08934]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.320943 15410 raft_consensus.cc:2726] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:01.310758 15441 raft_consensus.cc:482] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:01.288798 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0e69195596486fb03256baa22446e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.322536 15441 raft_consensus.cc:504] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:01.316397 13665 consensus_queue.cc:260] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:01.326647 15441 leader_election.cc:288] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:01.327316 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f280d8dbc048e5bcda6c5e9b2a7387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.353225 13274 tablet_service.cc:1467] Processing CreateTablet for tablet 8ce8e61126314f81a9b4d508971240b4 (DEFAULT_TABLE table=default.Table_98ae6d61510e4c97bea03912bdbbe8cc [id=88256e2f65354f26b2cd7bb8f0b1b7fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.355429 13274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ce8e61126314f81a9b4d508971240b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.347922 12237 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet c01ede4b52c3498a924340b39c3c2435 (table default.Table_ebe848b2f9ba4077a556369c87aa436a [id=ae7b5faec4c34d35981e0a1d464202ef]). Expected version 1 got 0
I20260810 23:33:01.362057 13691 ts_tablet_manager.cc:1428] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.627s user 0.012s sys 0.000s
I20260810 23:33:01.363785 12236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_d0b8422dca5e40ff919741fcee7d547c" } modify_external_catalogs: true
I20260810 23:33:01.364504 12236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_d0b8422dca5e40ff919741fcee7d547c" } modify_external_catalogs: true
I20260810 23:33:01.365032 13665 ts_tablet_manager.cc:1428] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.113s user 0.006s sys 0.005s
I20260810 23:33:01.367749 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3ac117f3b34c1bb55b6a258d4fd539" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:33:01.369959 12698 leader_election.cc:341] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:01.377733 12237 catalog_manager.cc:5495] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } }
I20260810 23:33:01.379854 13665 tablet_bootstrap.cc:492] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
W20260810 23:33:01.380656 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 151661us
I20260810 23:33:01.381944 13213 leader_election.cc:302] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
W20260810 23:33:01.362421 13691 ts_tablet_manager.cc:1460] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b: Tablet startup took 627ms
I20260810 23:33:01.387290 13514 tablet_bootstrap.cc:700] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent opening tablet: real 0.152s user 0.004s sys 0.000s
I20260810 23:33:01.389170 12755 tablet_service.cc:1467] Processing CreateTablet for tablet 8ce8e61126314f81a9b4d508971240b4 (DEFAULT_TABLE table=default.Table_98ae6d61510e4c97bea03912bdbbe8cc [id=88256e2f65354f26b2cd7bb8f0b1b7fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.391667 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ce8e61126314f81a9b4d508971240b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.266997 15433 tablet.cc:1722] T 217a7582902244f9b0ca65be1c20af22 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:33:01.400578 15217 raft_consensus.cc:2781] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:01.401233 15217 raft_consensus.cc:482] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:01.401775 15217 raft_consensus.cc:3037] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:33:01.386756 13691 ts_tablet_manager.cc:1462] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b: Trace:
0810 23:32:39.374488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:39.374692 (+ 204us) service_pool.cc:227] Handling call
0810 23:32:39.552813 (+178121us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:39.553026 (+ 213us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:39.853836 (+300810us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:40.329432 (+475596us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:40.330321 (+ 889us) inbound_call.cc:173] Queueing success response
0810 23:33:00.241746 (+19911425us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0810 23:33:00.734921 (+493175us) ts_tablet_manager.cc:1429] Starting tablet replica
0810 23:33:00.820163 (+ 85242us) tablet_replica.cc:259] Starting instrumentation
0810 23:33:00.820966 (+ 803us) tablet_replica.cc:278] Starting consensus
Metrics: {"mutex_wait_us":98658,"raft.queue_time_us":307593,"raft.run_cpu_time_us":469,"raft.run_wall_time_us":17372,"tablet-open.queue_time_us":19911572}
I20260810 23:33:01.414353 13691 tablet_bootstrap.cc:492] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:01.422632 15217 raft_consensus.cc:504] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:01.423451 12240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_bec275e3dbf54eb08d1df8d68e06b5a5" } modify_external_catalogs: true
I20260810 23:33:01.424419 12240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_bec275e3dbf54eb08d1df8d68e06b5a5" } modify_external_catalogs: true
I20260810 23:33:01.439450 12766 tablet_service.cc:1467] Processing CreateTablet for tablet 3a85273186994f2885a82ef2fe020ca4 (DEFAULT_TABLE table=default.Table_d54c3da29f874f2e83e5a10727950256 [id=f775b2636c034546ba65c9fc32fefecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.440927 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a85273186994f2885a82ef2fe020ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.289907 14121 ts_tablet_manager.cc:1918] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260810 23:33:01.452337 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 201185us
I20260810 23:33:01.442902 14121 log.cc:1196] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/ca0dd19112504f8ca57f64dcf0bd50a8
I20260810 23:33:01.455572 13514 tablet_bootstrap.cc:654] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:01.463354 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 32fad37a28e14903b8bde5aac6ee4e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:00 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:01.465600 13017 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd with delete_type TABLET_DATA_DELETED (replaced by tablet 46450c84a9054fbdbb5523ec835d43d2) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:01.465649 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 40eb874e0b334c46961e16d8d5aaaaa7 (DEFAULT_TABLE table=default.Table_de805fc39d9948e1a6c78d2979ad6e73 [id=c5b82d7c6f8a4505a1058a2aafa45422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.466912 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40eb874e0b334c46961e16d8d5aaaaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.469002 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 7031c257fdf34a28b6954d3d4827a8da (DEFAULT_TABLE table=default.Table_f64ee2d577744a919f65f54c28ccc88b [id=ca7378a0859a467dbd7c50eff469e96d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.470325 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7031c257fdf34a28b6954d3d4827a8da. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:01.460916 15421 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.346s user 0.004s sys 0.006s
I20260810 23:33:01.473791 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0e69195596486fb03256baa22446e0 (DEFAULT_TABLE table=default.Table_763854d208924ff78919903a71a7c826 [id=575025603ae44b0785de76cc5a28ab9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.475672 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0e69195596486fb03256baa22446e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.480119 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240adfa4be844518bbc9624d0e408789" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:01.490321 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_abf0a10256d44150b985a6d480568080"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:01.494604 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abf0a10256d44150b985a6d480568080 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:01.504936 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41ddf9f6f71443ba75c97d2b010372c" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:01.502395 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62705f99687340aaa1aca0942b28487f" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:01.523010 13017 tablet_service.cc:1467] Processing CreateTablet for tablet d3f280d8dbc048e5bcda6c5e9b2a7387 (DEFAULT_TABLE table=default.Table_d9fd2a26e54c483ebbc082b5a4ce014f [id=deb4795f344f4f108092842f7bd08934]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.525202 13014 tablet_service.cc:1467] Processing CreateTablet for tablet 8ce8e61126314f81a9b4d508971240b4 (DEFAULT_TABLE table=default.Table_98ae6d61510e4c97bea03912bdbbe8cc [id=88256e2f65354f26b2cd7bb8f0b1b7fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.528502 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f280d8dbc048e5bcda6c5e9b2a7387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.516577 14121 ts_tablet_manager.cc:1939] T ca0dd19112504f8ca57f64dcf0bd50a8 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:33:01.531037 13277 tablet_service.cc:1467] Processing CreateTablet for tablet 3a85273186994f2885a82ef2fe020ca4 (DEFAULT_TABLE table=default.Table_d54c3da29f874f2e83e5a10727950256 [id=f775b2636c034546ba65c9fc32fefecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.533007 13278 tablet_service.cc:1467] Processing CreateTablet for tablet 40eb874e0b334c46961e16d8d5aaaaa7 (DEFAULT_TABLE table=default.Table_de805fc39d9948e1a6c78d2979ad6e73 [id=c5b82d7c6f8a4505a1058a2aafa45422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.534797 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40eb874e0b334c46961e16d8d5aaaaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.536986 12789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62705f99687340aaa1aca0942b28487f" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:01.537695 12789 raft_consensus.cc:2454] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:01.539299 13665 tablet_bootstrap.cc:654] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:01.542335 13277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a85273186994f2885a82ef2fe020ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.543612 15439 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:33:01.549304 13031 tablet_service.cc:1467] Processing CreateTablet for tablet 40eb874e0b334c46961e16d8d5aaaaa7 (DEFAULT_TABLE table=default.Table_de805fc39d9948e1a6c78d2979ad6e73 [id=c5b82d7c6f8a4505a1058a2aafa45422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.550985 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40eb874e0b334c46961e16d8d5aaaaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.550755 13025 tablet_service.cc:1467] Processing CreateTablet for tablet 3a85273186994f2885a82ef2fe020ca4 (DEFAULT_TABLE table=default.Table_d54c3da29f874f2e83e5a10727950256 [id=f775b2636c034546ba65c9fc32fefecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:01.552946 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ce8e61126314f81a9b4d508971240b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:01.556641 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a85273186994f2885a82ef2fe020ca4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:01.578518 13212 leader_election.cc:341] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:01.598650 13212 leader_election.cc:341] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:01.599870 13212 leader_election.cc:341] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:01.628154 12228 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6f53aca31ee4e338d9b5e281557c121 consensus state has no leader
I20260810 23:33:01.643487 12236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 88c53785fdd24df79f138df6176e75d2
I20260810 23:33:01.648741 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c53785fdd24df79f138df6176e75d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:01 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:01.654785 15134 raft_consensus.cc:482] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f54aa78a60784a3bb2f5eb4bb2401ce3)
I20260810 23:33:01.655758 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 225) took 9311 ms. Trace:
I20260810 23:33:01.656561 14121 rpcz_store.cc:276] 0810 23:32:52.344528 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.344701 (+ 173us) service_pool.cc:227] Handling call
0810 23:33:00.112196 (+7767495us) ts_tablet_manager.cc:1145] Deleting tablet ca0dd19112504f8ca57f64dcf0bd50a8
0810 23:33:00.207874 (+ 95678us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:00.452793 (+244919us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b54000b27d4. stack: 00007fa86ed56033 00007fa86ed55f30 00007fa86e8d7534 00000000004db2b4 00000000004db249 00007fa875c62471 00007fa875ce5352 00007fa8760fb26c 00007fa876d08cfa 00007fa876d38e10 00007fa876d1fe0d 00007fa876d1fdb9 00007fa876d1fd49 00007fa876d1fd11 00007fa876d1ef3d 0000000000580784
0810 23:33:00.491199 (+ 38406us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b58000f38b0. stack: 00007fa86ed56033 00007fa86ed55f30 00007fa86e8d7534 00000000004db2b4 00000000004db249 00007fa875c62471 00007fa875c58f63 00007fa875cbcaba 00007fa875ce5375 00007fa8760fb26c 00007fa876d08cfa 00007fa876d38e10 00007fa876d1fe0d 00007fa876d1fdb9 00007fa876d1fd49 00007fa876d1fd11
0810 23:33:00.859947 (+368748us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:01.289567 (+429620us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:01.655640 (+366073us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":225091,"spinlock_wait_cycles":144820352,"tablet-delete.queue_time_us":7758903}
I20260810 23:33:01.659198 15217 leader_election.cc:288] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:01.659562 15455 raft_consensus.cc:482] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:01.675212 15391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:58598 (request call id 278) took 1608 ms. Trace:
I20260810 23:33:01.684087 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240adfa4be844518bbc9624d0e408789" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:01.655598 15134 raft_consensus.cc:504] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:01.691299 15134 leader_election.cc:288] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:01.700462 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3ac117f3b34c1bb55b6a258d4fd539" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:01.709635 13691 tablet_bootstrap.cc:654] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
W20260810 23:33:01.709240 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 108350us
I20260810 23:33:01.715818 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c53785fdd24df79f138df6176e75d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:01 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:33:01.716471 13212 consensus_peers.cc:487] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Couldn't send request to peer f26517150f1c480a877f15aec71d020b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:01.716728 15473 log.cc:924] Time spent T cd24471dd52149d395808493ad9fa68c P 191d120f4c7f44fc8475305a6d2e75f3: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
W20260810 23:33:01.717458 13212 leader_election.cc:341] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:01.722627 12958 leader_election.cc:341] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 92b81e7056a441d2854ba5ab6b4c5433
I20260810 23:33:01.681461 15455 raft_consensus.cc:504] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:01.716850 15217 consensus_queue.cc:1035] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260810 23:33:01.729616 13059 raft_consensus.cc:3037] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:01.732952 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 226) took 9360 ms. Trace:
I20260810 23:33:01.733328 14121 rpcz_store.cc:276] 0810 23:32:52.373048 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:52.373716 (+ 668us) service_pool.cc:227] Handling call
0810 23:33:01.658771 (+9285055us) ts_tablet_manager.cc:1145] Deleting tablet d603569c54ef4ffba9941d0cc3e0f97f
0810 23:33:01.658888 (+ 117us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:01.732875 (+ 73987us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9284162}
I20260810 23:33:01.736542 13406 heartbeater.cc:347] Connected to a master server at 127.11.226.189:42101
I20260810 23:33:01.737605 13406 heartbeater.cc:503] Master 127.11.226.189:42101 was elected leader, sending a full tablet report...
I20260810 23:33:01.740182 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet ca0dd19112504f8ca57f64dcf0bd50a8 (table default.Table_1e5f07908d444cf9af2cae2ea83ae7ae [id=a591a34da3514b87aa852684669a399f]) successfully deleted
I20260810 23:33:01.750444 13665 tablet_bootstrap.cc:492] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:01.767982 13514 tablet_bootstrap.cc:492] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:33:01.768658 13514 ts_tablet_manager.cc:1397] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.684s user 0.012s sys 0.005s
I20260810 23:33:01.770313 12226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_c5b6c13bda814cb98726f32aee432aab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:01.781271 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_b48833820bf944b688ae9585fb4daf68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:01.783773 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b48833820bf944b688ae9585fb4daf68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:01.785027 12226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5b6c13bda814cb98726f32aee432aab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:01.818912 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet d603569c54ef4ffba9941d0cc3e0f97f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d603569c54ef4ffba9941d0cc3e0f97f already in progress: creating tablet
I20260810 23:33:01.822124 15455 leader_election.cc:288] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:01.824083 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 238) took 7452 ms. Trace:
I20260810 23:33:01.824617 14121 rpcz_store.cc:276] 0810 23:32:54.371129 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:54.371488 (+ 359us) service_pool.cc:227] Handling call
0810 23:33:01.776935 (+7405447us) ts_tablet_manager.cc:1145] Deleting tablet f0cba350e63e46a894a89ccc5727aab8
0810 23:33:01.786810 (+ 9875us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:01.823858 (+ 37048us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7361652}
I20260810 23:33:01.826331 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 237) took 7459 ms. Trace:
I20260810 23:33:01.826655 14121 rpcz_store.cc:276] 0810 23:32:54.367319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:54.368330 (+ 1011us) service_pool.cc:227] Handling call
0810 23:33:01.825614 (+7457284us) ts_tablet_manager.cc:1145] Deleting tablet 0df95e01ce894651a0f49396baf49c51
0810 23:33:01.825735 (+ 121us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:01.826268 (+ 533us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7447849}
I20260810 23:33:01.675839 15391 rpcz_store.cc:276] 0810 23:33:00.067076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.067301 (+ 225us) service_pool.cc:227] Handling call
0810 23:33:01.675036 (+1607735us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:00.116340 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 23:33:00.160308 (+ 43968us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 23:33:00.160384 (+ 76us) alter_schema_op.cc:65] Acquired schema lock
0810 23:33:00.160519 (+ 135us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 23:33:00.160708 (+ 189us) alter_schema_op.cc:126] START. Timestamp: 1786404780160041
0810 23:33:00.160761 (+ 53us) op_driver.cc:331] REPLICATION: starting
0810 23:33:00.161778 (+ 1017us) log.cc:842] Serialized 173 byte log entry
0810 23:33:00.615283 (+453505us) op_driver.cc:444] REPLICATION: finished
0810 23:33:00.802144 (+186861us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 23:33:00.945768 (+143624us) log.cc:842] Serialized 5 byte log entry
0810 23:33:01.550764 (+604996us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:01.674759 (+123995us) log.cc:842] Serialized 14 byte log entry
0810 23:33:01.674874 (+ 115us) alter_schema_op.cc:71] Released schema lock
0810 23:33:01.674897 (+ 23us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":186233,"mutex_wait_us":2891,"prepare.queue_time_us":40097,"prepare.run_cpu_time_us":15436,"prepare.run_wall_time_us":223245,"raft.queue_time_us":4044,"raft.run_cpu_time_us":983,"raft.run_wall_time_us":982,"replication_time_us":454426,"thread_start_us":246940,"threads_started":2,"wal-append.queue_time_us":69216}]]}
I20260810 23:33:01.864358 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 239) took 6855 ms. Trace:
I20260810 23:33:01.865190 14121 rpcz_store.cc:276] 0810 23:32:55.009006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:55.009159 (+ 153us) service_pool.cc:227] Handling call
0810 23:33:01.827710 (+6818551us) ts_tablet_manager.cc:1145] Deleting tablet 4f453bd72168429086837b2b9b09fc78
0810 23:33:01.827792 (+ 82us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:01.864237 (+ 36445us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6811616}
I20260810 23:33:01.872346 12240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 8163921fdc2c442cb335041c79de6d6d
I20260810 23:33:01.876163 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_8371cab4a7ff44ceb3b02e4cb333fae1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:01.877032 12764 tablet_service.cc:1514] Processing DeleteTablet for tablet 8163921fdc2c442cb335041c79de6d6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:01 UTC) from {username='slave'} at 127.0.0.1:47546
W20260810 23:33:01.872961 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 124681us
W20260810 23:33:01.878255 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8371cab4a7ff44ceb3b02e4cb333fae1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:01.827982 15486 tablet_replica.cc:330] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:01.883422 15486 raft_consensus.cc:2229] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:01.884523 15486 raft_consensus.cc:2258] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:01.887073 15486 ts_tablet_manager.cc:1905] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:01.913110 15082 ts_tablet_manager.cc:1918] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:01.913988 15082 log.cc:1196] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/f1e8311c941b43058384cee6d8263dc1
I20260810 23:33:01.915037 15082 ts_tablet_manager.cc:1939] T f1e8311c941b43058384cee6d8263dc1 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:33:01.929312 12226 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_e2972dea79e44002b43aac315db921ca" } new_table_name: "default.Table_7e1011c9a03343acb3bdf04b07eb962e" modify_external_catalogs: true
I20260810 23:33:01.936501 12234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_de805fc39d9948e1a6c78d2979ad6e73" } modify_external_catalogs: true
I20260810 23:33:01.937124 12234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_de805fc39d9948e1a6c78d2979ad6e73" } modify_external_catalogs: true
I20260810 23:33:01.964704 13514 raft_consensus.cc:348] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:01.965940 13665 ts_tablet_manager.cc:1397] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.586s user 0.008s sys 0.014s
I20260810 23:33:01.966137 13514 raft_consensus.cc:374] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:01.966554 13514 raft_consensus.cc:729] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:33:01.967515 13514 consensus_queue.cc:260] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:01.983579 12236 master_service.cc:946] ReplaceTablet: received request to replace tablet dfda619f19a5459494b1e4f6d22821cc from {username='slave'} at 127.0.0.1:34742
I20260810 23:33:02.006428 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet 8163921fdc2c442cb335041c79de6d6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:01 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:02.007531 13010 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c53785fdd24df79f138df6176e75d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:01 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:02.017798 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 245) took 5334 ms. Trace:
W20260810 23:33:02.018424 12226 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6f53aca31ee4e338d9b5e281557c121 consensus state has no leader
I20260810 23:33:02.018414 14121 rpcz_store.cc:276] 0810 23:32:56.683557 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:56.683688 (+ 131us) service_pool.cc:227] Handling call
0810 23:33:01.984120 (+5300432us) ts_tablet_manager.cc:1145] Deleting tablet 5d96bc27a87a41908f3ec838c228030d
0810 23:33:01.993276 (+ 9156us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:02.017727 (+ 24451us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5299645}
W20260810 23:33:02.027437 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 275565us
W20260810 23:33:02.030200 12952 consensus_peers.cc:487] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f3049bcdc6e43c795961c74f0d008f5. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:02.003134 15465 log.cc:924] Time spent T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
W20260810 23:33:02.049373 12952 leader_election.cc:341] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 92b81e7056a441d2854ba5ab6b4c5433
W20260810 23:33:02.006299 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet f0cba350e63e46a894a89ccc5727aab8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0cba350e63e46a894a89ccc5727aab8 already in progress: creating tablet
W20260810 23:33:02.058459 12215 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 52211us
W20260810 23:33:02.059574 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 0df95e01ce894651a0f49396baf49c51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0df95e01ce894651a0f49396baf49c51 already in progress: creating tablet
W20260810 23:33:02.060483 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 4f453bd72168429086837b2b9b09fc78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f453bd72168429086837b2b9b09fc78 already in progress: creating tablet
I20260810 23:33:01.969105 15481 tablet.cc:1722] T c01ede4b52c3498a924340b39c3c2435 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:33:01.971066 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 272) took 4311 ms. Trace:
I20260810 23:33:02.067270 15082 rpcz_store.cc:276] 0810 23:32:57.659724 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.671913 (+ 12189us) service_pool.cc:227] Handling call
0810 23:33:00.892061 (+3220148us) ts_tablet_manager.cc:1145] Deleting tablet f1e8311c941b43058384cee6d8263dc1
0810 23:33:00.892121 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:00.940477 (+ 48356us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b54001707d4. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3c5352 00007f3cde7db26c 00007f3cdf3e8cfa 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49 00007f3cdf3ffd11 00007f3cdf3fef3d 0000000000580784
0810 23:33:01.143377 (+202900us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:01.912664 (+769287us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:01.970984 (+ 58320us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21991,"spinlock_wait_cycles":63433472,"tablet-delete.queue_time_us":3218908}
I20260810 23:33:02.069238 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 276) took 4239 ms. Trace:
I20260810 23:33:02.069585 15082 rpcz_store.cc:276] 0810 23:32:57.830093 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.903024 (+ 72931us) service_pool.cc:227] Handling call
0810 23:33:02.068813 (+4165789us) ts_tablet_manager.cc:1145] Deleting tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea
0810 23:33:02.068871 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:02.069171 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4147887}
I20260810 23:33:02.070703 15082 tablet_replica.cc:330] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:33:02.071676 15082 raft_consensus.cc:2229] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260810 23:33:01.889561 12757 consensus_meta.cc:287] T 7031c257fdf34a28b6954d3d4827a8da P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.602s user 0.009s sys 0.002s
I20260810 23:33:01.985018 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0136a1e890436fae572b46e64750e7" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:02.004174 15410 raft_consensus.cc:482] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:33:02.003811 12700 leader_election.cc:341] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:02.022732 15490 tablet.cc:1722] T bd77526f1db4455291fde8a001092a09 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:33:02.074064 12700 leader_election.cc:302] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:33:02.074839 13212 consensus_peers.cc:487] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Couldn't send request to peer f26517150f1c480a877f15aec71d020b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:02.075512 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet f1e8311c941b43058384cee6d8263dc1 (table default.Table_d8373b61098245bcaab43448a20578f6 [id=7934cc21dac24c74848a668abe1dd606]) successfully deleted
W20260810 23:33:02.076380 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea already in progress: creating tablet
I20260810 23:33:02.074436 15410 raft_consensus.cc:504] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:02.077778 13213 leader_election.cc:302] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:33:02.079193 15082 raft_consensus.cc:2258] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:02.079435 15410 leader_election.cc:288] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
W20260810 23:33:02.081045 13213 leader_election.cc:341] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: a41ddf9f6f71443ba75c97d2b010372c
I20260810 23:33:02.081524 13213 leader_election.cc:302] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:02.084489 15082 ts_tablet_manager.cc:1905] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:02.087116 12226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_f64ee2d577744a919f65f54c28ccc88b" } modify_external_catalogs: true
I20260810 23:33:02.087909 12226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_f64ee2d577744a919f65f54c28ccc88b" } modify_external_catalogs: true
W20260810 23:33:02.075519 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 71761us
W20260810 23:33:02.093381 12700 leader_election.cc:341] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:02.093832 12700 leader_election.cc:302] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:02.095454 15498 raft_consensus.cc:2726] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:02.075886 15488 raft_consensus.cc:2726] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:33:02.097218 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 5d96bc27a87a41908f3ec838c228030d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d96bc27a87a41908f3ec838c228030d already in progress: creating tablet
W20260810 23:33:02.097739 12700 leader_election.cc:341] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:02.050058 12952 leader_election.cc:302] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:02.098169 12700 leader_election.cc:302] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:02.098943 15488 raft_consensus.cc:2726] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:33:02.099054 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 49724us
I20260810 23:33:02.101967 14121 tablet_replica.cc:330] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:33:02.105790 12797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240adfa4be844518bbc9624d0e408789" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:02.109300 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 8163921fdc2c442cb335041c79de6d6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:01 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:02.109540 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0136a1e890436fae572b46e64750e7" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:02.110970 13691 tablet_bootstrap.cc:492] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:02.111481 13691 ts_tablet_manager.cc:1397] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.697s user 0.012s sys 0.014s
W20260810 23:33:02.119733 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet a6f53aca31ee4e338d9b5e281557c121 (table default.Table_7e1011c9a03343acb3bdf04b07eb962e [id=89c842ab64c546bbb967a2246a7220ac], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet a6f53aca31ee4e338d9b5e281557c121 consensus state has no leader
I20260810 23:33:02.121754 12878 heartbeater.cc:347] Connected to a master server at 127.11.226.189:42101
I20260810 23:33:02.137708 13514 ts_tablet_manager.cc:1428] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.369s user 0.008s sys 0.000s
I20260810 23:33:02.115105 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8c0d1dacc344cbacb0c5988f0e2221" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:02.139441 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_88ec9ab54a584cd996eac5baed70b544"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:02.141368 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88ec9ab54a584cd996eac5baed70b544 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:02.142076 12700 leader_election.cc:341] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:02.148151 15445 raft_consensus.cc:2726] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:33:02.150346 14121 raft_consensus.cc:2229] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:02.159099 14121 raft_consensus.cc:2258] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:02.159771 13141 heartbeater.cc:347] Connected to a master server at 127.11.226.189:42101
I20260810 23:33:02.162254 15510 raft_consensus.cc:482] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:02.163077 15510 raft_consensus.cc:504] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:02.161120 15490 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:47546 (request call id 296) took 1138 ms. Trace:
I20260810 23:33:02.176098 13514 tablet_bootstrap.cc:492] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:33:02.183068 13018 consensus_meta.cc:287] T 7031c257fdf34a28b6954d3d4827a8da P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.546s user 0.010s sys 0.000s
I20260810 23:33:02.186599 15439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Bootstrap starting.
I20260810 23:33:02.203841 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 12b8279474454c73ac935a7ccd004407 (DEFAULT_TABLE table=default.Table_a8f723dad5a54fceb62622c2aefe987c [id=35b68d151d0b47a5a90c0c4e1e762d74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.204241 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdd421bc27d46e39c6491ea7ac0f3bf (DEFAULT_TABLE table=default.Table_90b34e5db74b40c1954de96c5c9fa3d1 [id=0bf66ae0183340f9b3394ad6b0ac5c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.205256 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b8279474454c73ac935a7ccd004407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.205377 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdd421bc27d46e39c6491ea7ac0f3bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.207875 12224 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet c01ede4b52c3498a924340b39c3c2435 (table default.Table_ebe848b2f9ba4077a556369c87aa436a [id=ae7b5faec4c34d35981e0a1d464202ef]). Expected version 1 got 0
I20260810 23:33:02.212476 12234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 40eb874e0b334c46961e16d8d5aaaaa7
I20260810 23:33:02.215663 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet 40eb874e0b334c46961e16d8d5aaaaa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:01 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:02.218452 12234 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_9cab5b899a3748378d28d31c84e8b090"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:02.221983 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 138155us
W20260810 23:33:02.224521 12234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cab5b899a3748378d28d31c84e8b090 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:02.172178 15490 rpcz_store.cc:276] 0810 23:33:01.023104 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:01.023255 (+ 151us) service_pool.cc:227] Handling call
0810 23:33:02.161036 (+1137781us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:01.024865 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 23:33:01.044098 (+ 19233us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 23:33:01.044145 (+ 47us) alter_schema_op.cc:65] Acquired schema lock
0810 23:33:01.044232 (+ 87us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 23:33:01.044386 (+ 154us) alter_schema_op.cc:126] START. Timestamp: 1786404781044282
0810 23:33:01.044412 (+ 26us) op_driver.cc:331] REPLICATION: starting
0810 23:33:01.045083 (+ 671us) log.cc:842] Serialized 173 byte log entry
0810 23:33:01.753442 (+708359us) op_driver.cc:444] REPLICATION: finished
0810 23:33:02.022611 (+269169us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 23:33:02.076754 (+ 54143us) log.cc:842] Serialized 5 byte log entry
0810 23:33:02.150961 (+ 74207us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:02.157640 (+ 6679us) log.cc:842] Serialized 14 byte log entry
0810 23:33:02.160869 (+ 3229us) alter_schema_op.cc:71] Released schema lock
0810 23:33:02.160897 (+ 28us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":268887,"prepare.queue_time_us":473,"prepare.run_cpu_time_us":15180,"prepare.run_wall_time_us":236297,"replication_time_us":708909,"thread_start_us":484262,"threads_started":2,"wal-append.queue_time_us":194942}]]}
I20260810 23:33:02.238471 15510 leader_election.cc:288] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:02.243428 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet 40eb874e0b334c46961e16d8d5aaaaa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:01 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:02.246130 12226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 7031c257fdf34a28b6954d3d4827a8da
I20260810 23:33:02.254663 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet 7031c257fdf34a28b6954d3d4827a8da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:02 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:02.266919 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet 7031c257fdf34a28b6954d3d4827a8da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:02 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:02.283265 13031 tablet_service.cc:1467] Processing CreateTablet for tablet 12b8279474454c73ac935a7ccd004407 (DEFAULT_TABLE table=default.Table_a8f723dad5a54fceb62622c2aefe987c [id=35b68d151d0b47a5a90c0c4e1e762d74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.284899 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b8279474454c73ac935a7ccd004407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.257284 13665 raft_consensus.cc:348] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:02.298296 13665 raft_consensus.cc:374] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:02.298746 13665 raft_consensus.cc:729] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:02.299517 13665 consensus_queue.cc:260] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:02.301405 13665 ts_tablet_manager.cc:1428] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.335s user 0.005s sys 0.004s
I20260810 23:33:02.259017 15373 raft_consensus.cc:2781] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:02.259866 15217 raft_consensus.cc:2726] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:02.302253 15373 raft_consensus.cc:482] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:02.261009 15460 consensus_queue.cc:1035] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:33:02.302659 15373 raft_consensus.cc:3037] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:02.261085 15466 consensus_queue.cc:1035] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:33:02.219662 12224 catalog_manager.cc:5495] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } }
I20260810 23:33:02.308051 12224 catalog_manager.cc:5495] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } }
I20260810 23:33:02.309819 12224 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5]). Expected version 2 got 0
I20260810 23:33:02.310871 12224 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (table default.Table_4a7751c5b61c4332adeeb9ee00474b27 [id=2c8320255aab4036b5350d082f9e04ef]). Expected version 2 got 0
I20260810 23:33:02.313421 13665 tablet_bootstrap.cc:492] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
W20260810 23:33:02.323671 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 100738us
I20260810 23:33:02.329627 12236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet dfda619f19a5459494b1e4f6d22821cc
I20260810 23:33:02.291793 12952 leader_election.cc:302] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:02.331207 12236 catalog_manager.cc:6565] ReplaceTablet: tablet dfda619f19a5459494b1e4f6d22821cc deleted and replaced by tablet 5135df28790d42d2be48053436c6cc60
I20260810 23:33:02.335952 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet dfda619f19a5459494b1e4f6d22821cc with delete_type TABLET_DATA_DELETED (replaced by tablet 5135df28790d42d2be48053436c6cc60) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:02.337215 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_566f96b5dead41b78406bb2776c7121b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:02.342959 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_566f96b5dead41b78406bb2776c7121b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:02.343812 12755 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdd421bc27d46e39c6491ea7ac0f3bf (DEFAULT_TABLE table=default.Table_90b34e5db74b40c1954de96c5c9fa3d1 [id=0bf66ae0183340f9b3394ad6b0ac5c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.349828 13514 tablet_bootstrap.cc:654] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:02.353574 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdd421bc27d46e39c6491ea7ac0f3bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.355252 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_76308997ea5c4140ac84b26ccc7cc1bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:02.357631 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76308997ea5c4140ac84b26ccc7cc1bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:02.357456 13010 tablet_service.cc:1467] Processing CreateTablet for tablet 19f5d509a67741fe97dd041b2dda9d75 (DEFAULT_TABLE table=default.Table_94d5b6058b6041b79fb4ac9e4e07f354 [id=5fd2e68f76b441b1854a99c3eaad5596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.358863 13010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f5d509a67741fe97dd041b2dda9d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.363466 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet dfda619f19a5459494b1e4f6d22821cc with delete_type TABLET_DATA_DELETED (replaced by tablet 5135df28790d42d2be48053436c6cc60) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:02.368389 12764 tablet_service.cc:1467] Processing CreateTablet for tablet 19f5d509a67741fe97dd041b2dda9d75 (DEFAULT_TABLE table=default.Table_94d5b6058b6041b79fb4ac9e4e07f354 [id=5fd2e68f76b441b1854a99c3eaad5596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.369587 12764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f5d509a67741fe97dd041b2dda9d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.389979 13023 tablet_service.cc:1467] Processing CreateTablet for tablet c6b6dc7d429e4d68876545645e4ba4c0 (DEFAULT_TABLE table=default.Table_4ad6c4068abe4c64a425bea91494c59a [id=a6ac59b4e35043e1b236644dca9a5c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.391752 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b6dc7d429e4d68876545645e4ba4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.393563 12768 tablet_service.cc:1467] Processing CreateTablet for tablet c6b6dc7d429e4d68876545645e4ba4c0 (DEFAULT_TABLE table=default.Table_4ad6c4068abe4c64a425bea91494c59a [id=a6ac59b4e35043e1b236644dca9a5c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.394719 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b6dc7d429e4d68876545645e4ba4c0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:02.413116 13213 leader_election.cc:341] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: Tablet not RUNNING: STOPPED
I20260810 23:33:02.413631 13213 leader_election.cc:302] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:02.415580 15082 ts_tablet_manager.cc:1918] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:02.418555 14121 ts_tablet_manager.cc:1905] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:02.427521 15217 raft_consensus.cc:2726] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:33:02.428452 13691 raft_consensus.cc:348] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:33:02.429109 12952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 95873.3us
I20260810 23:33:02.431856 15217 consensus_queue.cc:1035] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:33:02.435286 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8c0d1dacc344cbacb0c5988f0e2221" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:02.428385 15455 raft_consensus.cc:2726] T 0a0136a1e890436fae572b46e64750e7 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:02.432411 15082 log.cc:1196] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/240adfa4be844518bbc9624d0e408789
I20260810 23:33:02.323380 15486 ts_tablet_manager.cc:1918] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260810 23:33:02.441975 15486 log.cc:1196] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/88c53785fdd24df79f138df6176e75d2
I20260810 23:33:02.443872 15082 ts_tablet_manager.cc:1939] T 240adfa4be844518bbc9624d0e408789 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:33:02.445152 12797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b9d20d09c4d1ba9a4b10959807e50" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:02.449193 12797 raft_consensus.cc:2454] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
W20260810 23:33:02.445957 12698 leader_election.cc:341] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:02.450253 12698 leader_election.cc:302] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:02.452193 15410 raft_consensus.cc:2726] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:02.459257 15486 ts_tablet_manager.cc:1939] T 88c53785fdd24df79f138df6176e75d2 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:02.463809 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 281) took 3501 ms. Trace:
I20260810 23:33:02.467432 13665 tablet_bootstrap.cc:654] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:02.448434 15373 raft_consensus.cc:504] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:02.476974 15082 rpcz_store.cc:276] 0810 23:32:58.962558 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:58.962725 (+ 167us) service_pool.cc:227] Handling call
0810 23:33:02.070494 (+3107769us) ts_tablet_manager.cc:1145] Deleting tablet 240adfa4be844518bbc9624d0e408789
0810 23:33:02.070558 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:02.206799 (+136241us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:02.415253 (+208454us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:02.463730 (+ 48477us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3106771}
I20260810 23:33:02.477778 15373 leader_election.cc:288] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:02.479773 13213 leader_election.cc:302] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
W20260810 23:33:02.480751 13213 consensus_peers.cc:487] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e8311c941b43058384cee6d8263dc1. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:02.470631 15523 tablet.cc:1722] T bd77526f1db4455291fde8a001092a09 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:33:02.484635 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 282) took 3519 ms. Trace:
I20260810 23:33:02.484840 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 240adfa4be844518bbc9624d0e408789 (table default.Table_32d7e9316d054deca361c6e3d9b5e1f4 [id=43ef82e91d4a4e2b8bb06d4cab61c8c4]) successfully deleted
I20260810 23:33:02.487468 15518 raft_consensus.cc:2781] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:02.487962 15518 raft_consensus.cc:482] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:02.488365 15518 raft_consensus.cc:3037] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:02.499696 12234 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_5cde9c2beb0448c1971328536e2c47d8" } new_table_name: "default.Table_8a358dec4da84beda7fdf1eb485eb587" modify_external_catalogs: true
I20260810 23:33:02.500924 15518 raft_consensus.cc:504] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:02.429287 13691 raft_consensus.cc:374] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:02.503023 15518 leader_election.cc:288] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:02.503062 13691 raft_consensus.cc:729] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:02.504292 15082 rpcz_store.cc:276] 0810 23:32:58.965167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:58.965314 (+ 147us) service_pool.cc:227] Handling call
0810 23:33:02.484323 (+3519009us) ts_tablet_manager.cc:1145] Deleting tablet 4cdec13ee70d402fa3a08119e7bf1313
0810 23:33:02.484376 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:02.484570 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3517958}
W20260810 23:33:02.505834 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 4cdec13ee70d402fa3a08119e7bf1313 because the tablet was not found. No further retry: Not found: Tablet not found: 4cdec13ee70d402fa3a08119e7bf1313
W20260810 23:33:02.507591 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 364858us
I20260810 23:33:02.517445 12763 tablet_service.cc:1467] Processing CreateTablet for tablet a72d6225d41e45ee86e00efd7270e44a (DEFAULT_TABLE table=default.Table_1ebd777d240743bbb5739b63d7e00b39 [id=4acb486be7454f6093439e084454ed36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.524449 15082 tablet_replica.cc:330] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:33:02.525472 15082 raft_consensus.cc:2229] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:02.526944 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72d6225d41e45ee86e00efd7270e44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.537223 12240 master_service.cc:946] ReplaceTablet: received request to replace tablet f47d8e462d814a15a82fd0d5218e2d0e from {username='slave'} at 127.0.0.1:34742
I20260810 23:33:02.537178 13691 consensus_queue.cc:260] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:02.546499 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b9d20d09c4d1ba9a4b10959807e50" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:02.556260 15373 raft_consensus.cc:482] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:02.556972 15373 raft_consensus.cc:504] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:02.559628 15373 leader_election.cc:288] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:02.577316 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62705f99687340aaa1aca0942b28487f" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:02.579401 15461 raft_consensus.cc:482] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:02.580080 15461 raft_consensus.cc:504] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:02.582768 15461 leader_election.cc:288] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:02.585580 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b9d20d09c4d1ba9a4b10959807e50" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:02.589231 13277 tablet_service.cc:1467] Processing CreateTablet for tablet 12b8279474454c73ac935a7ccd004407 (DEFAULT_TABLE table=default.Table_a8f723dad5a54fceb62622c2aefe987c [id=35b68d151d0b47a5a90c0c4e1e762d74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.590979 13277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b8279474454c73ac935a7ccd004407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.592805 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 40eb874e0b334c46961e16d8d5aaaaa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:01 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:02.597255 15439 log.cc:824] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:02.599094 13691 ts_tablet_manager.cc:1428] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.487s user 0.014s sys 0.009s
W20260810 23:33:02.600814 15522 log.cc:924] Time spent T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b: Append to log took a long time: real 0.053s user 0.003s sys 0.000s
I20260810 23:33:02.605181 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 7031c257fdf34a28b6954d3d4827a8da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:02 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:02.609709 13665 tablet_bootstrap.cc:492] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:02.610394 13665 ts_tablet_manager.cc:1397] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.297s user 0.009s sys 0.005s
I20260810 23:33:02.614370 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet dfda619f19a5459494b1e4f6d22821cc with delete_type TABLET_DATA_DELETED (replaced by tablet 5135df28790d42d2be48053436c6cc60) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:02.621621 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdd421bc27d46e39c6491ea7ac0f3bf (DEFAULT_TABLE table=default.Table_90b34e5db74b40c1954de96c5c9fa3d1 [id=0bf66ae0183340f9b3394ad6b0ac5c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.622900 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdd421bc27d46e39c6491ea7ac0f3bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.623059 13014 tablet_service.cc:1467] Processing CreateTablet for tablet a72d6225d41e45ee86e00efd7270e44a (DEFAULT_TABLE table=default.Table_1ebd777d240743bbb5739b63d7e00b39 [id=4acb486be7454f6093439e084454ed36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.624318 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72d6225d41e45ee86e00efd7270e44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.627599 12236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_d8a0e3e655b9417d9df3ff6d82b76f15" } modify_external_catalogs: true
I20260810 23:33:02.628276 12236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_d8a0e3e655b9417d9df3ff6d82b76f15" } modify_external_catalogs: true
I20260810 23:33:02.635404 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf4ff4b1264442b8ca1dbde64583220 (DEFAULT_TABLE table=default.Table_e156205c664d4b45b29a0e258e39cf34 [id=ba18db75ae0b49b4ac6246e838459efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.636744 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf4ff4b1264442b8ca1dbde64583220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.637259 13278 tablet_service.cc:1467] Processing CreateTablet for tablet 19f5d509a67741fe97dd041b2dda9d75 (DEFAULT_TABLE table=default.Table_94d5b6058b6041b79fb4ac9e4e07f354 [id=5fd2e68f76b441b1854a99c3eaad5596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.638616 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f5d509a67741fe97dd041b2dda9d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.648473 13691 tablet_bootstrap.cc:492] T 684caf5aaa8347e8b4e02323318fb0b7 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:02.649889 13274 tablet_service.cc:1467] Processing CreateTablet for tablet c6b6dc7d429e4d68876545645e4ba4c0 (DEFAULT_TABLE table=default.Table_4ad6c4068abe4c64a425bea91494c59a [id=a6ac59b4e35043e1b236644dca9a5c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.651273 13274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b6dc7d429e4d68876545645e4ba4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.651417 13017 tablet_service.cc:1467] Processing CreateTablet for tablet a29bcea786e744b689c63c611ec0d944 (DEFAULT_TABLE table=default.Table_c764bc5f83fe4b25aaff52cac5e1009e [id=5b473a1a3aff4ff9adfd4a0542cab809]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.654336 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29bcea786e744b689c63c611ec0d944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.657666 13273 tablet_service.cc:1467] Processing CreateTablet for tablet a72d6225d41e45ee86e00efd7270e44a (DEFAULT_TABLE table=default.Table_1ebd777d240743bbb5739b63d7e00b39 [id=4acb486be7454f6093439e084454ed36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.659096 13273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72d6225d41e45ee86e00efd7270e44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.668756 12766 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf4ff4b1264442b8ca1dbde64583220 (DEFAULT_TABLE table=default.Table_e156205c664d4b45b29a0e258e39cf34 [id=ba18db75ae0b49b4ac6246e838459efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.670054 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf4ff4b1264442b8ca1dbde64583220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.681097 13665 raft_consensus.cc:348] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:02.682266 13665 raft_consensus.cc:374] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:02.683115 13665 raft_consensus.cc:729] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:02.694593 13025 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd999bbcc04431f826ffccf360fc56d (DEFAULT_TABLE table=default.Table_be4c5de03d7649a08296569cbe6a3503 [id=77ef566f33a94336b6f94c0a117312a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.695847 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd999bbcc04431f826ffccf360fc56d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:02.701006 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 249064us
I20260810 23:33:02.703083 15082 raft_consensus.cc:2258] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:02.702885 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf4ff4b1264442b8ca1dbde64583220 (DEFAULT_TABLE table=default.Table_e156205c664d4b45b29a0e258e39cf34 [id=ba18db75ae0b49b4ac6246e838459efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.704005 13283 tablet_service.cc:1467] Processing CreateTablet for tablet a29bcea786e744b689c63c611ec0d944 (DEFAULT_TABLE table=default.Table_c764bc5f83fe4b25aaff52cac5e1009e [id=5b473a1a3aff4ff9adfd4a0542cab809]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.705076 15082 ts_tablet_manager.cc:1905] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:02.705276 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29bcea786e744b689c63c611ec0d944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.710067 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf4ff4b1264442b8ca1dbde64583220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.713007 12757 tablet_service.cc:1467] Processing CreateTablet for tablet a29bcea786e744b689c63c611ec0d944 (DEFAULT_TABLE table=default.Table_c764bc5f83fe4b25aaff52cac5e1009e [id=5b473a1a3aff4ff9adfd4a0542cab809]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.713426 13279 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd999bbcc04431f826ffccf360fc56d (DEFAULT_TABLE table=default.Table_be4c5de03d7649a08296569cbe6a3503 [id=77ef566f33a94336b6f94c0a117312a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.714424 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29bcea786e744b689c63c611ec0d944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.714540 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd999bbcc04431f826ffccf360fc56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.746093 12761 tablet_service.cc:1467] Processing CreateTablet for tablet d94f4f33fce54d7bacd0af1c28249398 (DEFAULT_TABLE table=default.Table_1de5219e712b481d8048407bd4cec52a [id=7cffa285f3a04f74a80ec7c12b7b3c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.747565 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd999bbcc04431f826ffccf360fc56d (DEFAULT_TABLE table=default.Table_be4c5de03d7649a08296569cbe6a3503 [id=77ef566f33a94336b6f94c0a117312a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.748835 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd999bbcc04431f826ffccf360fc56d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:02.755146 15439 log.cc:924] Time spent T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Append to log took a long time: real 0.055s user 0.004s sys 0.000s
I20260810 23:33:02.756647 14121 ts_tablet_manager.cc:1918] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:02.757248 14121 log.cc:1196] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/92b81e7056a441d2854ba5ab6b4c5433
I20260810 23:33:02.758373 14121 ts_tablet_manager.cc:1939] T 92b81e7056a441d2854ba5ab6b4c5433 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:33:02.738039 12785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62705f99687340aaa1aca0942b28487f" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
W20260810 23:33:02.769222 13212 leader_election.cc:341] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:02.683984 13665 consensus_queue.cc:260] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:02.831674 15488 raft_consensus.cc:482] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:02.832441 15488 raft_consensus.cc:504] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:02.836614 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94f4f33fce54d7bacd0af1c28249398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.840937 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b9d20d09c4d1ba9a4b10959807e50" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:02.841674 12803 raft_consensus.cc:3037] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:02.850155 15488 leader_election.cc:288] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:02.853098 13514 tablet_bootstrap.cc:492] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:33:02.853856 13514 ts_tablet_manager.cc:1397] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.678s user 0.018s sys 0.006s
I20260810 23:33:02.875795 13665 ts_tablet_manager.cc:1428] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.265s user 0.005s sys 0.002s
I20260810 23:33:02.885426 13012 tablet_service.cc:1467] Processing CreateTablet for tablet d94f4f33fce54d7bacd0af1c28249398 (DEFAULT_TABLE table=default.Table_1de5219e712b481d8048407bd4cec52a [id=7cffa285f3a04f74a80ec7c12b7b3c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.894805 15440 raft_consensus.cc:482] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:02.895692 15440 raft_consensus.cc:504] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:02.901396 15440 leader_election.cc:288] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:02.905668 12760 tablet_service.cc:1467] Processing CreateTablet for tablet 7af2a9ae2d1e4939889b1ac87ec0d154 (DEFAULT_TABLE table=default.Table_714aaba90fb0449bb9a5cb6ac00bae8c [id=6a20c65432e44766a6572a26e1336116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.907580 12760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af2a9ae2d1e4939889b1ac87ec0d154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.909574 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 255) took 5573 ms. Trace:
I20260810 23:33:02.910030 14121 rpcz_store.cc:276] 0810 23:32:57.336551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.337960 (+ 1409us) service_pool.cc:227] Handling call
0810 23:33:02.047849 (+4709889us) ts_tablet_manager.cc:1145] Deleting tablet 92b81e7056a441d2854ba5ab6b4c5433
0810 23:33:02.095551 (+ 47702us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:02.153739 (+ 58188us) spinlock_profiling.cc:243] Waited 47.4 ms on lock 0x7b54001216d4. stack: 00007fa86ed56033 00007fa86ed55f30 00007fa86e8d7534 00000000004db2b4 00000000004db249 00007fa875c62471 00007fa875ce5352 00007fa8760fb26c 00007fa876d08cfa 00007fa876d38e10 00007fa876d1fe0d 00007fa876d1fdb9 00007fa876d1fd49 00007fa876d1fd11 00007fa876d1ef3d 0000000000580784
0810 23:33:02.662996 (+509257us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:02.752086 (+ 89090us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:02.909473 (+157387us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26214,"spinlock_wait_cycles":108990208,"tablet-delete.queue_time_us":4679782}
W20260810 23:33:02.877034 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 96097.3us
I20260810 23:33:02.924304 13012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94f4f33fce54d7bacd0af1c28249398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.912062 15439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=2 applied=8 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:33:02.928565 15439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Bootstrap complete.
I20260810 23:33:02.929386 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 257) took 5400 ms. Trace:
I20260810 23:33:02.929893 14121 rpcz_store.cc:276] 0810 23:32:57.528657 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.528820 (+ 163us) service_pool.cc:227] Handling call
0810 23:33:02.911812 (+5382992us) ts_tablet_manager.cc:1145] Deleting tablet 133654008f5247b2ac87dcd09195f197
0810 23:33:02.911928 (+ 116us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:02.929290 (+ 17362us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5274432}
I20260810 23:33:02.930084 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684caf5aaa8347e8b4e02323318fb0b7" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:02.932212 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 258) took 5062 ms. Trace:
W20260810 23:33:02.932459 13212 leader_election.cc:341] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:02.932696 13691 tablet_bootstrap.cc:654] T 684caf5aaa8347e8b4e02323318fb0b7 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
W20260810 23:33:02.933284 13212 leader_election.cc:341] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:02.937348 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 88c53785fdd24df79f138df6176e75d2 (table default.Table_d0b8422dca5e40ff919741fcee7d547c [id=71c9424f327040a68889fc257935c535]) successfully deleted
I20260810 23:33:02.931666 13327 raft_consensus.cc:2454] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
W20260810 23:33:02.947608 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 8163921fdc2c442cb335041c79de6d6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8163921fdc2c442cb335041c79de6d6d already in progress: creating tablet
W20260810 23:33:02.950378 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 40eb874e0b334c46961e16d8d5aaaaa7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40eb874e0b334c46961e16d8d5aaaaa7 already in progress: creating tablet
W20260810 23:33:02.952395 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 7031c257fdf34a28b6954d3d4827a8da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7031c257fdf34a28b6954d3d4827a8da already in progress: creating tablet
I20260810 23:33:02.956502 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 7af2a9ae2d1e4939889b1ac87ec0d154 (DEFAULT_TABLE table=default.Table_714aaba90fb0449bb9a5cb6ac00bae8c [id=6a20c65432e44766a6572a26e1336116]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:33:02.956743 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet dfda619f19a5459494b1e4f6d22821cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfda619f19a5459494b1e4f6d22821cc already in progress: creating tablet
I20260810 23:33:02.957784 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af2a9ae2d1e4939889b1ac87ec0d154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.932572 14121 rpcz_store.cc:276] 0810 23:32:57.870362 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.870528 (+ 166us) service_pool.cc:227] Handling call
0810 23:33:02.931579 (+5061051us) ts_tablet_manager.cc:1145] Deleting tablet 42517f11422b42c39db282e2c5f584a3
0810 23:33:02.931725 (+ 146us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:02.932154 (+ 429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5060266}
I20260810 23:33:02.978137 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 259) took 5107 ms. Trace:
I20260810 23:33:02.978216 12240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet f47d8e462d814a15a82fd0d5218e2d0e
I20260810 23:33:02.978521 14121 rpcz_store.cc:276] 0810 23:32:57.870891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.871377 (+ 486us) service_pool.cc:227] Handling call
0810 23:33:02.977753 (+5106376us) ts_tablet_manager.cc:1145] Deleting tablet 081b4e1f53c74f768a4bf401ae22b585
0810 23:33:02.977836 (+ 83us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:02.978072 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5105793}
I20260810 23:33:02.979636 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 260) took 5108 ms. Trace:
I20260810 23:33:02.979909 12240 catalog_manager.cc:6565] ReplaceTablet: tablet f47d8e462d814a15a82fd0d5218e2d0e deleted and replaced by tablet 0703e531759b4709bd3917cd9d307dd2
W20260810 23:33:02.982761 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 318095us
I20260810 23:33:02.985008 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet f47d8e462d814a15a82fd0d5218e2d0e with delete_type TABLET_DATA_DELETED (replaced by tablet 0703e531759b4709bd3917cd9d307dd2) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:02.986105 13406 heartbeater.cc:503] Master 127.11.226.189:42101 was elected leader, sending a full tablet report...
I20260810 23:33:02.987042 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 92b81e7056a441d2854ba5ab6b4c5433 (table default.Table_e0faeb2fda9f4af48212613473c88681 [id=d8d3c66123cd4516a59a0ab8ae4d47b9]) successfully deleted
W20260810 23:33:02.987797 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 133654008f5247b2ac87dcd09195f197 because the tablet was not found. No further retry: Not found: Tablet not found: 133654008f5247b2ac87dcd09195f197
W20260810 23:33:02.989271 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 42517f11422b42c39db282e2c5f584a3 because the tablet was not found. No further retry: Not found: Tablet not found: 42517f11422b42c39db282e2c5f584a3
I20260810 23:33:02.988538 13010 tablet_service.cc:1467] Processing CreateTablet for tablet 46450c84a9054fbdbb5523ec835d43d2 (DEFAULT_TABLE table=default.Table_9625301c8cda460981e80e753c785d4b [id=d79b564767e24f3b947e21b4e56e2e3b]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:33:02.990258 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 081b4e1f53c74f768a4bf401ae22b585 because the tablet was not found. No further retry: Not found: Tablet not found: 081b4e1f53c74f768a4bf401ae22b585
I20260810 23:33:02.990837 13010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46450c84a9054fbdbb5523ec835d43d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.990715 13286 tablet_service.cc:1467] Processing CreateTablet for tablet d94f4f33fce54d7bacd0af1c28249398 (DEFAULT_TABLE table=default.Table_1de5219e712b481d8048407bd4cec52a [id=7cffa285f3a04f74a80ec7c12b7b3c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.993006 13014 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb2cfdf755a4dfaa1ea8a30ccb51045 (DEFAULT_TABLE table=default.Table_b78ae79905d8448085a6e318220c1584 [id=b0b48348e1b444bbb9ac2d2cdeb078d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:02.994433 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb2cfdf755a4dfaa1ea8a30ccb51045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:02.997345 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 7af2a9ae2d1e4939889b1ac87ec0d154 (DEFAULT_TABLE table=default.Table_714aaba90fb0449bb9a5cb6ac00bae8c [id=6a20c65432e44766a6572a26e1336116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:03.001886 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb2cfdf755a4dfaa1ea8a30ccb51045 (DEFAULT_TABLE table=default.Table_b78ae79905d8448085a6e318220c1584 [id=b0b48348e1b444bbb9ac2d2cdeb078d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:03.005573 13275 tablet_service.cc:1514] Processing DeleteTablet for tablet f47d8e462d814a15a82fd0d5218e2d0e with delete_type TABLET_DATA_DELETED (replaced by tablet 0703e531759b4709bd3917cd9d307dd2) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:03.006278 13280 tablet_service.cc:1467] Processing CreateTablet for tablet 46450c84a9054fbdbb5523ec835d43d2 (DEFAULT_TABLE table=default.Table_9625301c8cda460981e80e753c785d4b [id=d79b564767e24f3b947e21b4e56e2e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:03.011922 12240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_c5b6c13bda814cb98726f32aee432aab" } modify_external_catalogs: true
I20260810 23:33:03.012768 12240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_c5b6c13bda814cb98726f32aee432aab" } modify_external_catalogs: true
W20260810 23:33:03.037581 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 148086us
I20260810 23:33:03.043555 12226 catalog_manager.cc:5495] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } }
I20260810 23:33:03.046859 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (table default.Table_4a7751c5b61c4332adeeb9ee00474b27 [id=2c8320255aab4036b5350d082f9e04ef]). Expected version 2 got 0
I20260810 23:33:03.048076 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]). Expected version 1 got 0
I20260810 23:33:02.979944 14121 rpcz_store.cc:276] 0810 23:32:57.871260 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.872173 (+ 913us) service_pool.cc:227] Handling call
0810 23:33:02.979322 (+5107149us) ts_tablet_manager.cc:1145] Deleting tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e
0810 23:33:02.979376 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:02.979578 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5106615}
I20260810 23:33:03.050082 12226 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5]). Expected version 2 got 0
I20260810 23:33:03.054827 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 262) took 5183 ms. Trace:
I20260810 23:33:03.055303 14121 rpcz_store.cc:276] 0810 23:32:57.871882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.872057 (+ 175us) service_pool.cc:227] Handling call
0810 23:33:03.049057 (+5177000us) ts_tablet_manager.cc:1145] Deleting tablet b261673faa964b65af3576c89e302864
0810 23:33:03.049954 (+ 897us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:03.054754 (+ 4800us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5176220}
W20260810 23:33:03.063498 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet d30bc9bd4e1f462b8d3be7f8a2f28f1e because the tablet was not found. No further retry: Not found: Tablet not found: d30bc9bd4e1f462b8d3be7f8a2f28f1e
W20260810 23:33:03.064730 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet b261673faa964b65af3576c89e302864 because the tablet was not found. No further retry: Not found: Tablet not found: b261673faa964b65af3576c89e302864
I20260810 23:33:03.059976 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684caf5aaa8347e8b4e02323318fb0b7" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:03.060868 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1f24e226b94ccf8ad5a656c80a5b3b" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:03.071382 13141 heartbeater.cc:347] Connected to a master server at 127.11.226.189:42101
I20260810 23:33:03.082228 13059 raft_consensus.cc:3037] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:03.056556 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 263) took 5184 ms. Trace:
I20260810 23:33:03.098454 13691 tablet_bootstrap.cc:492] T 684caf5aaa8347e8b4e02323318fb0b7 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:03.100329 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1452c292e9c476b9e26f56998f0ad55" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:33:03.105724 13212 leader_election.cc:341] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:03.106465 13212 leader_election.cc:341] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:03.118178 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46450c84a9054fbdbb5523ec835d43d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.118410 13665 tablet_bootstrap.cc:492] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:03.133443 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb2cfdf755a4dfaa1ea8a30ccb51045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.136090 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94f4f33fce54d7bacd0af1c28249398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.139859 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af2a9ae2d1e4939889b1ac87ec0d154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.092419 14121 rpcz_store.cc:276] 0810 23:32:57.872272 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.873239 (+ 967us) service_pool.cc:227] Handling call
0810 23:33:03.056227 (+5182988us) ts_tablet_manager.cc:1145] Deleting tablet fd4f55ef709d473f89a16d32160eb61b
0810 23:33:03.056291 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:03.056500 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5181844}
I20260810 23:33:03.141793 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 261) took 5270 ms. Trace:
I20260810 23:33:03.142061 15082 ts_tablet_manager.cc:1918] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:03.142501 12698 leader_election.cc:302] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:03.143356 15410 raft_consensus.cc:2781] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:03.143723 15410 raft_consensus.cc:482] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:03.144059 15410 raft_consensus.cc:3037] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:33:03.146296 12698 leader_election.cc:341] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:03.147629 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet f47d8e462d814a15a82fd0d5218e2d0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f47d8e462d814a15a82fd0d5218e2d0e already in progress: creating tablet
I20260810 23:33:03.171054 12240 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet da5411e9a70844f891f620d66c732247
I20260810 23:33:03.181465 13665 tablet_bootstrap.cc:654] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:03.183008 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_9eec1fcf95c04f239f4265b71add01c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:03.184830 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9eec1fcf95c04f239f4265b71add01c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:03.195315 12767 tablet_service.cc:1514] Processing DeleteTablet for tablet f47d8e462d814a15a82fd0d5218e2d0e with delete_type TABLET_DATA_DELETED (replaced by tablet 0703e531759b4709bd3917cd9d307dd2) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:03.212857 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb2cfdf755a4dfaa1ea8a30ccb51045 (DEFAULT_TABLE table=default.Table_b78ae79905d8448085a6e318220c1584 [id=b0b48348e1b444bbb9ac2d2cdeb078d9]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:33:03.216187 12217 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet a6f53aca31ee4e338d9b5e281557c121 (table default.Table_7e1011c9a03343acb3bdf04b07eb962e [id=89c842ab64c546bbb967a2246a7220ac], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet a6f53aca31ee4e338d9b5e281557c121 consensus state has no leader
I20260810 23:33:03.202941 13514 raft_consensus.cc:348] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:03.218061 13514 raft_consensus.cc:374] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:03.218508 13514 raft_consensus.cc:729] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:33:03.142539 15082 log.cc:1196] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/62705f99687340aaa1aca0942b28487f
I20260810 23:33:03.219944 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684caf5aaa8347e8b4e02323318fb0b7" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:03.223433 13514 consensus_queue.cc:260] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:03.227360 15082 ts_tablet_manager.cc:1939] T 62705f99687340aaa1aca0942b28487f P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:33:03.239089 13665 tablet_bootstrap.cc:492] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:03.239717 13665 ts_tablet_manager.cc:1397] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.121s user 0.014s sys 0.000s
I20260810 23:33:03.255237 13665 raft_consensus.cc:348] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:03.261831 15461 raft_consensus.cc:482] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:03.266228 15461 raft_consensus.cc:504] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:03.273041 15461 leader_election.cc:288] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:03.255998 13665 raft_consensus.cc:374] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:03.273995 13665 raft_consensus.cc:729] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:03.283980 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a480093c481941f0bb35949c7606f37f" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:33:03.285362 13212 leader_election.cc:341] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:03.285099 13665 consensus_queue.cc:260] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:03.287865 13665 ts_tablet_manager.cc:1428] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.048s user 0.004s sys 0.004s
W20260810 23:33:03.215482 12755 consensus_meta.cc:287] T 8fdd421bc27d46e39c6491ea7ac0f3bf P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.852s user 0.007s sys 0.009s
I20260810 23:33:03.222535 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 46450c84a9054fbdbb5523ec835d43d2 (DEFAULT_TABLE table=default.Table_9625301c8cda460981e80e753c785d4b [id=d79b564767e24f3b947e21b4e56e2e3b]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:33:03.259413 12797 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.807s user 0.010s sys 0.026s
W20260810 23:33:03.320577 12797 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.868s user 0.011s sys 0.026s
I20260810 23:33:03.283110 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb2cfdf755a4dfaa1ea8a30ccb51045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.329576 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46450c84a9054fbdbb5523ec835d43d2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:03.334744 13213 leader_election.cc:341] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: Tablet not RUNNING: STOPPED
I20260810 23:33:03.335263 13213 leader_election.cc:302] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:03.337427 15373 raft_consensus.cc:2726] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:33:03.324147 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 283) took 4346 ms. Trace:
I20260810 23:33:03.338043 15082 rpcz_store.cc:276] 0810 23:32:58.977054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:58.977188 (+ 134us) service_pool.cc:227] Handling call
0810 23:33:02.516353 (+3539165us) ts_tablet_manager.cc:1145] Deleting tablet 62705f99687340aaa1aca0942b28487f
0810 23:33:02.516409 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:02.805881 (+289472us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:03.079894 (+274013us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:03.323807 (+243913us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":160211,"tablet-delete.queue_time_us":3538417}
I20260810 23:33:03.339954 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684caf5aaa8347e8b4e02323318fb0b7" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:03.342695 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet bd77526f1db4455291fde8a001092a09 (table default.Table_970b3cb3161f4f5a9fec0f3dfa72f1a5 [id=963512beef0d4063b8e6a5738abd1d92]). Expected version 3 got 2
I20260810 23:33:03.337986 15439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:33:03.345327 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_02075f5963604d95b8d962f1071ebe64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:03.348068 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02075f5963604d95b8d962f1071ebe64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:03.350847 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1452c292e9c476b9e26f56998f0ad55" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:03.287125 12803 raft_consensus.cc:2454] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:33:03.371057 15082 tablet_replica.cc:330] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:33:03.372011 15082 raft_consensus.cc:2229] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:03.315613 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a480093c481941f0bb35949c7606f37f" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:03.377107 12805 raft_consensus.cc:2454] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:03.343781 12234 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_8a358dec4da84beda7fdf1eb485eb587 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 5 got 4
I20260810 23:33:03.343634 15439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198f534261994047a68e1d509bb2fd8b, State: Initialized, Role: FOLLOWER
I20260810 23:33:03.383119 15439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:33:03.390889 15410 raft_consensus.cc:504] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:03.394706 12806 raft_consensus.cc:2379] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in current term 1: Already voted for candidate 191d120f4c7f44fc8475305a6d2e75f3 in this term.
I20260810 23:33:03.402135 15082 raft_consensus.cc:2258] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:03.404994 15082 ts_tablet_manager.cc:1905] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:03.405010 15570 tablet.cc:1722] T bd77526f1db4455291fde8a001092a09 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260810 23:33:03.408718 15568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:33:03.409869 15568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:03.412328 12235 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2] alter complete (version 1)
I20260810 23:33:03.413149 12236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet ab8d987109ba40198645881b449cb972
I20260810 23:33:03.414752 15439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:33:03.432937 12223 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 337) took 3673 ms. Trace:
I20260810 23:33:03.433344 12223 rpcz_store.cc:276] 0810 23:32:59.759522 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.964781 (+205259us) service_pool.cc:227] Handling call
0810 23:33:00.000085 (+ 35304us) spinlock_profiling.cc:243] Waited 33.5 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e77d6 00007f0ce6e2d634 00007f0ce6e2d5c2 00007f0ce6e2d4f1 00007f0ce6e2d46c 00007f0ce6e2c722 00007f0ce50c694c 00007f0ce50c5d86
0810 23:33:00.024693 (+ 24608us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:33:00.076275 (+ 51582us) catalog_manager.cc:3616] Apply alter schema
0810 23:33:00.077085 (+ 810us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:33:00.077399 (+ 314us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:33:00.406934 (+329535us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:33:03.432795 (+3025861us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:00.251700 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:00.252330 (+ 630us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:00.252364 (+ 34us) write_op.cc:435] Acquired schema lock
0810 23:33:00.252381 (+ 17us) tablet.cc:660] Decoding operations
0810 23:33:00.253067 (+ 686us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:00.253118 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:00.253143 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:00.253412 (+ 269us) tablet.cc:699] Row locks acquired
0810 23:33:00.253439 (+ 27us) write_op.cc:260] PREPARE: finished
0810 23:33:00.253745 (+ 306us) write_op.cc:270] Start()
0810 23:33:00.253907 (+ 162us) write_op.cc:276] Timestamp: P: 1786404780253641 usec, L: 0
0810 23:33:00.253933 (+ 26us) op_driver.cc:331] REPLICATION: starting
0810 23:33:00.255731 (+ 1798us) log.cc:842] Serialized 405 byte log entry
0810 23:33:00.404658 (+148927us) op_driver.cc:444] REPLICATION: finished
0810 23:33:00.405375 (+ 717us) write_op.cc:301] APPLY: starting
0810 23:33:00.405448 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:00.405573 (+ 125us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:00.405750 (+ 177us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:00.405950 (+ 200us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:00.406121 (+ 171us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:00.406172 (+ 51us) write_op.cc:312] APPLY: finished
0810 23:33:00.406413 (+ 241us) log.cc:842] Serialized 23 byte log entry
0810 23:33:00.406599 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:00.406671 (+ 72us) write_op.cc:454] Released schema lock
0810 23:33:00.406703 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:00.406903 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:00.406924 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":23316,"spinlock_wait_cycles":77155072,"child_traces":[["op",{"apply.queue_time_us":503,"apply.run_cpu_time_us":1717,"apply.run_wall_time_us":1720,"num_ops":1,"prepare.queue_time_us":172715,"prepare.run_cpu_time_us":3323,"prepare.run_wall_time_us":6340,"replication_time_us":150606}]]}
I20260810 23:33:03.285070 13514 ts_tablet_manager.cc:1428] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.431s user 0.007s sys 0.001s
W20260810 23:33:03.448665 13514 ts_tablet_manager.cc:1460] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3: Tablet startup took 594ms
W20260810 23:33:03.449234 13514 ts_tablet_manager.cc:1462] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3: Trace:
0810 23:32:43.216672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:43.216870 (+ 198us) service_pool.cc:227] Handling call
0810 23:32:43.713600 (+496730us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:43.713716 (+ 116us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:44.150010 (+436294us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:44.531715 (+381705us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:44.536364 (+ 4649us) inbound_call.cc:173] Queueing success response
0810 23:33:02.175714 (+17639350us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0810 23:33:02.854374 (+678660us) ts_tablet_manager.cc:1429] Starting tablet replica
0810 23:33:03.200757 (+346383us) tablet_replica.cc:259] Starting instrumentation
0810 23:33:03.201490 (+ 733us) tablet_replica.cc:278] Starting consensus
Metrics: {"mutex_wait_us":153,"raft.queue_time_us":52351,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":939,"tablet-open.queue_time_us":17639449}
I20260810 23:33:03.452302 13514 tablet_bootstrap.cc:492] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:33:03.462491 12226 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_ebe848b2f9ba4077a556369c87aa436a [id=ae7b5faec4c34d35981e0a1d464202ef] alter complete (version 1)
W20260810 23:33:03.468061 12229 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 43) took 2753 ms (client timeout 1000 ms). Trace:
W20260810 23:33:03.469429 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet fd4f55ef709d473f89a16d32160eb61b because the tablet was not found. No further retry: Not found: Tablet not found: fd4f55ef709d473f89a16d32160eb61b
I20260810 23:33:03.471112 13010 tablet_service.cc:1514] Processing DeleteTablet for tablet ab8d987109ba40198645881b449cb972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:02 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:03.473099 13010 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0dd19112504f8ca57f64dcf0bd50a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:40 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:03.473739 13014 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e8311c941b43058384cee6d8263dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:03.474423 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b81e7056a441d2854ba5ab6b4c5433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:55 UTC) from {username='slave'} at 127.0.0.1:58598
W20260810 23:33:03.471506 12229 rpcz_store.cc:269] 0810 23:33:00.714864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.750890 (+ 36026us) service_pool.cc:227] Handling call
0810 23:33:03.467946 (+2717056us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:01.224216 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:01.225108 (+ 892us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:01.225156 (+ 48us) write_op.cc:435] Acquired schema lock
0810 23:33:01.225198 (+ 42us) tablet.cc:660] Decoding operations
0810 23:33:01.225934 (+ 736us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:01.226004 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:01.226060 (+ 56us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:01.226389 (+ 329us) tablet.cc:699] Row locks acquired
0810 23:33:01.226420 (+ 31us) write_op.cc:260] PREPARE: finished
0810 23:33:01.259484 (+ 33064us) write_op.cc:270] Start()
0810 23:33:01.280190 (+ 20706us) write_op.cc:276] Timestamp: P: 1786404781259361 usec, L: 0
0810 23:33:01.280254 (+ 64us) op_driver.cc:331] REPLICATION: starting
0810 23:33:01.281404 (+ 1150us) log.cc:842] Serialized 536 byte log entry
0810 23:33:01.319061 (+ 37657us) op_driver.cc:444] REPLICATION: finished
0810 23:33:01.333634 (+ 14573us) write_op.cc:301] APPLY: starting
0810 23:33:01.333748 (+ 114us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:01.333921 (+ 173us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:01.334096 (+ 175us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:01.334408 (+ 312us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:01.334768 (+ 360us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:01.334840 (+ 72us) write_op.cc:312] APPLY: finished
0810 23:33:01.335178 (+ 338us) log.cc:842] Serialized 23 byte log entry
0810 23:33:01.335497 (+ 319us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:01.335628 (+ 131us) write_op.cc:454] Released schema lock
0810 23:33:01.335690 (+ 62us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:01.335999 (+ 309us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:01.336062 (+ 63us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":14284,"apply.run_cpu_time_us":2719,"apply.run_wall_time_us":2719,"num_ops":1,"prepare.queue_time_us":190794,"prepare.run_cpu_time_us":4612,"prepare.run_wall_time_us":57796,"raft.queue_time_us":10022,"raft.run_cpu_time_us":965,"raft.run_wall_time_us":74409,"replication_time_us":38696}]]}
I20260810 23:33:03.477475 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_98320d53af774eabb5d826d4c0732dad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:03.479310 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98320d53af774eabb5d826d4c0732dad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:03.484602 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_35fbbca76d6c43419846a96203e57279"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:03.487972 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 62705f99687340aaa1aca0942b28487f (table default.Table_96019e28037b4d38aa9fc933a29c8884 [id=dd7ca129aac0430ba386c53a30052148]) successfully deleted
I20260810 23:33:03.491034 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet ab8d987109ba40198645881b449cb972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:02 UTC) from {username='slave'} at 127.0.0.1:47546
W20260810 23:33:03.493774 12237 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 37) took 3943 ms (client timeout 1000 ms). Trace:
W20260810 23:33:03.495111 12237 rpcz_store.cc:269] 0810 23:32:59.550300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.585664 (+ 35364us) service_pool.cc:227] Handling call
0810 23:32:59.679059 (+ 93395us) spinlock_profiling.cc:243] Waited 73.2 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:33:03.493691 (+3814632us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:01.443974 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:01.444934 (+ 960us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:01.444985 (+ 51us) write_op.cc:435] Acquired schema lock
0810 23:33:01.444997 (+ 12us) tablet.cc:660] Decoding operations
0810 23:33:01.445604 (+ 607us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:01.445705 (+ 101us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:01.445753 (+ 48us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:01.446063 (+ 310us) tablet.cc:699] Row locks acquired
0810 23:33:01.446093 (+ 30us) write_op.cc:260] PREPARE: finished
0810 23:33:01.446363 (+ 270us) write_op.cc:270] Start()
0810 23:33:01.446547 (+ 184us) write_op.cc:276] Timestamp: P: 1786404781446256 usec, L: 0
0810 23:33:01.446565 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 23:33:01.447683 (+ 1118us) log.cc:842] Serialized 536 byte log entry
0810 23:33:01.722536 (+274853us) op_driver.cc:444] REPLICATION: finished
0810 23:33:01.786321 (+ 63785us) write_op.cc:301] APPLY: starting
0810 23:33:01.786429 (+ 108us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:01.786617 (+ 188us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:01.786762 (+ 145us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:01.787088 (+ 326us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:01.787335 (+ 247us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:01.787411 (+ 76us) write_op.cc:312] APPLY: finished
0810 23:33:01.787759 (+ 348us) log.cc:842] Serialized 23 byte log entry
0810 23:33:01.788767 (+ 1008us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:01.788893 (+ 126us) write_op.cc:454] Released schema lock
0810 23:33:01.788961 (+ 68us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:01.885668 (+ 96707us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:01.885726 (+ 58us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":18,"spinlock_wait_cycles":168376704,"child_traces":[["op",{"apply.queue_time_us":63489,"apply.run_cpu_time_us":2664,"apply.run_wall_time_us":99684,"mutex_wait_us":53026,"num_ops":1,"prepare.queue_time_us":39634,"prepare.run_cpu_time_us":4167,"prepare.run_wall_time_us":5684,"replication_time_us":275792}]]}
I20260810 23:33:03.502501 15082 ts_tablet_manager.cc:1918] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:33:03.515512 13691 ts_tablet_manager.cc:1397] T 684caf5aaa8347e8b4e02323318fb0b7 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.867s user 0.017s sys 0.015s
I20260810 23:33:03.520046 15082 log.cc:1196] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/f935119e4d9b4accb79db98185c80aca
I20260810 23:33:03.521445 15082 ts_tablet_manager.cc:1939] T f935119e4d9b4accb79db98185c80aca P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:33:03.522634 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 287) took 3928 ms. Trace:
I20260810 23:33:03.523240 15082 rpcz_store.cc:276] 0810 23:32:59.593836 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.594111 (+ 275us) service_pool.cc:227] Handling call
0810 23:33:03.370865 (+3776754us) ts_tablet_manager.cc:1145] Deleting tablet f935119e4d9b4accb79db98185c80aca
0810 23:33:03.370916 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:03.425551 (+ 54635us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:03.502214 (+ 76663us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:03.522567 (+ 20353us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3776120}
I20260810 23:33:03.524609 15082 tablet_replica.cc:330] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
W20260810 23:33:03.532794 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35fbbca76d6c43419846a96203e57279 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:03.142124 14121 rpcz_store.cc:276] 0810 23:32:57.871573 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.874602 (+ 3029us) service_pool.cc:227] Handling call
0810 23:33:03.141374 (+5266772us) ts_tablet_manager.cc:1145] Deleting tablet 08c45b71432a45269cd6a106fc5f7292
0810 23:33:03.141495 (+ 121us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:03.141728 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5265797}
I20260810 23:33:03.548097 12235 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_d028b6b163de469a9325b241cb5701a1 [id=49a3a44dd7f9473d9c025d8690a00c1f] alter complete (version 3)
I20260810 23:33:03.549582 15410 leader_election.cc:288] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:33:03.553737 12228 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 67) took 1844 ms (client timeout 1000 ms). Trace:
W20260810 23:33:03.556478 12228 rpcz_store.cc:269] 0810 23:33:01.709560 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:01.709749 (+ 189us) service_pool.cc:227] Handling call
0810 23:33:01.772766 (+ 63017us) spinlock_profiling.cc:243] Waited 59.5 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:33:03.553646 (+1780880us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":12,"spinlock_wait_cycles":136788224}
W20260810 23:33:03.560369 12235 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 66) took 2924 ms (client timeout 1000 ms). Trace:
W20260810 23:33:03.562623 12224 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 38) took 2727 ms (client timeout 1000 ms). Trace:
W20260810 23:33:03.564668 12700 leader_election.cc:341] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:03.560904 12235 rpcz_store.cc:269] 0810 23:33:00.636344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.636573 (+ 229us) service_pool.cc:227] Handling call
0810 23:33:03.560296 (+2923723us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:02.370870 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:02.371345 (+ 475us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:02.371375 (+ 30us) write_op.cc:435] Acquired schema lock
0810 23:33:02.371392 (+ 17us) tablet.cc:660] Decoding operations
0810 23:33:02.371703 (+ 311us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:02.371750 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:02.371775 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:02.371952 (+ 177us) tablet.cc:699] Row locks acquired
0810 23:33:02.371980 (+ 28us) write_op.cc:260] PREPARE: finished
0810 23:33:02.375020 (+ 3040us) write_op.cc:270] Start()
0810 23:33:02.375132 (+ 112us) write_op.cc:276] Timestamp: P: 1786404782374917 usec, L: 0
0810 23:33:02.375154 (+ 22us) op_driver.cc:331] REPLICATION: starting
0810 23:33:02.375737 (+ 583us) log.cc:842] Serialized 356 byte log entry
0810 23:33:02.390828 (+ 15091us) op_driver.cc:444] REPLICATION: finished
0810 23:33:02.391231 (+ 403us) write_op.cc:301] APPLY: starting
0810 23:33:02.391293 (+ 62us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:02.391409 (+ 116us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:02.391506 (+ 97us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:02.391778 (+ 272us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:02.391981 (+ 203us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:02.392082 (+ 101us) write_op.cc:312] APPLY: finished
0810 23:33:02.392310 (+ 228us) log.cc:842] Serialized 23 byte log entry
0810 23:33:02.392495 (+ 185us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:02.392567 (+ 72us) write_op.cc:454] Released schema lock
0810 23:33:02.392600 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:02.392790 (+ 190us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:02.392813 (+ 23us) write_op.cc:454] Released schema lock
Related trace 'op':
0810 23:33:03.443842 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:03.444280 (+ 438us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:03.444307 (+ 27us) write_op.cc:435] Acquired schema lock
0810 23:33:03.444326 (+ 19us) tablet.cc:660] Decoding operations
0810 23:33:03.444629 (+ 303us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:03.444673 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:03.444694 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:03.444847 (+ 153us) tablet.cc:699] Row locks acquired
0810 23:33:03.444864 (+ 17us) write_op.cc:260] PREPARE: finished
0810 23:33:03.444997 (+ 133us) write_op.cc:270] Start()
0810 23:33:03.445095 (+ 98us) write_op.cc:276] Timestamp: P: 1786404783444729 usec, L: 0
0810 23:33:03.445117 (+ 22us) op_driver.cc:331] REPLICATION: starting
0810 23:33:03.480678 (+ 35561us) log.cc:842] Serialized 356 byte log entry
0810 23:33:03.481928 (+ 1250us) spinlock_profiling.cc:243] Waited 34.9 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:33:03.541581 (+ 59653us) op_driver.cc:444] REPLICATION: finished
0810 23:33:03.541843 (+ 262us) write_op.cc:301] APPLY: starting
0810 23:33:03.541916 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:03.542032 (+ 116us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:03.542049 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:03.542244 (+ 195us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:03.542384 (+ 140us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:03.542432 (+ 48us) write_op.cc:312] APPLY: finished
0810 23:33:03.542636 (+ 204us) log.cc:842] Serialized 23 byte log entry
0810 23:33:03.542800 (+ 164us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:03.542875 (+ 75us) write_op.cc:454] Released schema lock
0810 23:33:03.542911 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:03.543142 (+ 231us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:03.543177 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":233,"apply.run_cpu_time_us":1746,"apply.run_wall_time_us":1745,"num_ops":1,"prepare.queue_time_us":88611,"prepare.run_cpu_time_us":2500,"prepare.run_wall_time_us":28874,"raft.queue_time_us":93,"raft.run_cpu_time_us":661,"raft.run_wall_time_us":663,"replication_time_us":15580}],["op",{"apply.queue_time_us":95,"apply.run_cpu_time_us":1546,"apply.run_wall_time_us":1545,"num_ops":1,"prepare.queue_time_us":109,"prepare.run_cpu_time_us":3824,"prepare.run_wall_time_us":38303,"raft.queue_time_us":13427,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":14,"replication_time_us":96364,"spinlock_wait_cycles":80261504}]]}
W20260810 23:33:03.569413 13213 leader_election.cc:341] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 1b1f24e226b94ccf8ad5a656c80a5b3b
I20260810 23:33:03.569883 13213 leader_election.cc:302] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:03.576768 12240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_9eec1fcf95c04f239f4265b71add01c2" } modify_external_catalogs: true
W20260810 23:33:03.576906 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 08c45b71432a45269cd6a106fc5f7292 because the tablet was not found. No further retry: Not found: Tablet not found: 08c45b71432a45269cd6a106fc5f7292
I20260810 23:33:03.577386 12240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_9eec1fcf95c04f239f4265b71add01c2" } modify_external_catalogs: true
I20260810 23:33:03.581055 12229 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_1ebd777d240743bbb5739b63d7e00b39" } new_table_name: "default.Table_cd3600f5c033449dae5897454ce350a7" modify_external_catalogs: true
I20260810 23:33:03.582808 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_64b9e1ba04bf4003bc78a1f24cdbd715"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:03.584389 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64b9e1ba04bf4003bc78a1f24cdbd715 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:03.587901 12700 leader_election.cc:341] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:03.588480 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]) successfully deleted
I20260810 23:33:03.588529 12700 leader_election.cc:302] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
W20260810 23:33:03.591773 12226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 39) took 1337 ms (client timeout 1000 ms). Trace:
I20260810 23:33:03.601202 12769 tablet_service.cc:1514] Processing DeleteTablet for tablet 806fe5162aef4d36ad5e2ac324047426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='slave'} at 127.0.0.1:47546
W20260810 23:33:03.600234 12226 rpcz_store.cc:269] 0810 23:33:02.254055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:02.256096 (+ 2041us) service_pool.cc:227] Handling call
0810 23:33:03.591482 (+1335386us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:03.188362 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:03.188894 (+ 532us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:03.188924 (+ 30us) write_op.cc:435] Acquired schema lock
0810 23:33:03.188945 (+ 21us) tablet.cc:660] Decoding operations
0810 23:33:03.189263 (+ 318us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:03.189311 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:03.189334 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:03.190596 (+ 1262us) tablet.cc:699] Row locks acquired
0810 23:33:03.190615 (+ 19us) write_op.cc:260] PREPARE: finished
0810 23:33:03.190744 (+ 129us) write_op.cc:270] Start()
0810 23:33:03.190835 (+ 91us) write_op.cc:276] Timestamp: P: 1786404783190498 usec, L: 0
0810 23:33:03.190853 (+ 18us) op_driver.cc:331] REPLICATION: starting
0810 23:33:03.191414 (+ 561us) log.cc:842] Serialized 536 byte log entry
0810 23:33:03.237210 (+ 45796us) op_driver.cc:444] REPLICATION: finished
0810 23:33:03.237700 (+ 490us) write_op.cc:301] APPLY: starting
0810 23:33:03.237773 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:03.237883 (+ 110us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:03.237991 (+ 108us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:03.238217 (+ 226us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:03.238383 (+ 166us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:03.238450 (+ 67us) write_op.cc:312] APPLY: finished
0810 23:33:03.238688 (+ 238us) log.cc:842] Serialized 23 byte log entry
0810 23:33:03.239695 (+ 1007us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:03.239775 (+ 80us) write_op.cc:454] Released schema lock
0810 23:33:03.239810 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:03.240144 (+ 334us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:03.240171 (+ 27us) write_op.cc:454] Released schema lock
Related trace 'op':
0810 23:33:03.339437 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:03.340055 (+ 618us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:03.340082 (+ 27us) write_op.cc:435] Acquired schema lock
0810 23:33:03.340095 (+ 13us) tablet.cc:660] Decoding operations
0810 23:33:03.340412 (+ 317us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:03.340458 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:03.340481 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:03.340639 (+ 158us) tablet.cc:699] Row locks acquired
0810 23:33:03.340653 (+ 14us) write_op.cc:260] PREPARE: finished
0810 23:33:03.340783 (+ 130us) write_op.cc:270] Start()
0810 23:33:03.374938 (+ 34155us) write_op.cc:276] Timestamp: P: 1786404783340522 usec, L: 0
0810 23:33:03.374965 (+ 27us) op_driver.cc:331] REPLICATION: starting
0810 23:33:03.375655 (+ 690us) log.cc:842] Serialized 356 byte log entry
0810 23:33:03.389408 (+ 13753us) op_driver.cc:444] REPLICATION: finished
0810 23:33:03.456261 (+ 66853us) write_op.cc:301] APPLY: starting
0810 23:33:03.456333 (+ 72us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:03.456444 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:03.456556 (+ 112us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:03.456749 (+ 193us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:03.456898 (+ 149us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:03.456944 (+ 46us) write_op.cc:312] APPLY: finished
0810 23:33:03.457162 (+ 218us) log.cc:842] Serialized 23 byte log entry
0810 23:33:03.457355 (+ 193us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:03.457419 (+ 64us) write_op.cc:454] Released schema lock
0810 23:33:03.457454 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:03.457645 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:03.457672 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":289,"apply.run_cpu_time_us":1788,"apply.run_wall_time_us":2675,"num_ops":1,"prepare.queue_time_us":120008,"prepare.run_cpu_time_us":2441,"prepare.run_wall_time_us":100791,"raft.queue_time_us":1906,"raft.run_cpu_time_us":845,"raft.run_wall_time_us":846,"replication_time_us":46258}],["op",{"apply.queue_time_us":66638,"apply.run_cpu_time_us":1588,"apply.run_wall_time_us":1588,"num_ops":1,"prepare.queue_time_us":93789,"prepare.run_cpu_time_us":2620,"prepare.run_wall_time_us":36565,"raft.queue_time_us":2003,"raft.run_cpu_time_us":773,"raft.run_wall_time_us":773,"replication_time_us":14356}]]}
W20260810 23:33:03.612746 12234 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 44) took 988 ms (client timeout 1000 ms). Trace:
W20260810 23:33:03.613248 12234 rpcz_store.cc:269] 0810 23:33:02.624792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:02.811531 (+186739us) service_pool.cc:227] Handling call
0810 23:33:02.955854 (+144323us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:33:03.612671 (+656817us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":24,"spinlock_wait_cycles":313970944}
I20260810 23:33:03.623520 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet ab8d987109ba40198645881b449cb972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:02 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:03.625000 15486 tablet_replica.cc:330] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:03.631430 13665 tablet_bootstrap.cc:492] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:03.631822 12226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_2ad7503d6e154e87a585df0b30bcbd1d" } modify_external_catalogs: true
I20260810 23:33:03.632457 12226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_2ad7503d6e154e87a585df0b30bcbd1d" } modify_external_catalogs: true
I20260810 23:33:03.638942 13514 tablet_bootstrap.cc:654] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:03.644054 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdec13ee70d402fa3a08119e7bf1313 with delete_type TABLET_DATA_DELETED (replaced by tablet d603569c54ef4ffba9941d0cc3e0f97f) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:03.608183 15410 raft_consensus.cc:2726] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:33:03.651094 12958 consensus_peers.cc:487] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:03.653975 15455 raft_consensus.cc:482] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f54aa78a60784a3bb2f5eb4bb2401ce3)
I20260810 23:33:03.657092 15455 raft_consensus.cc:504] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:03.664891 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet f47b8b57d5b44a018bc2d7b3f976d803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:03.669920 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet f935119e4d9b4accb79db98185c80aca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:58 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:03.628516 15373 raft_consensus.cc:482] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:03.670686 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e8311c941b43058384cee6d8263dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:03.671756 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 10033af79e854103bc47a046d2219ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:58 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:03.673566 15455 leader_election.cc:288] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:03.675426 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3049bcdc6e43c795961c74f0d008f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:03.677778 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 240adfa4be844518bbc9624d0e408789 with delete_type TABLET_DATA_DELETED (replaced by tablet 4344987a6c90428fbdf50765a4631455) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:03.681238 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 62705f99687340aaa1aca0942b28487f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:48 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:03.682389 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet a41ddf9f6f71443ba75c97d2b010372c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:03.685850 13665 tablet_bootstrap.cc:654] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:03.688913 15614 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:33:03.690513 15486 raft_consensus.cc:2229] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:03.691358 15486 raft_consensus.cc:2258] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:03.694598 15486 ts_tablet_manager.cc:1905] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:03.700893 15510 raft_consensus.cc:2726] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:03.673123 15373 raft_consensus.cc:504] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:03.706636 15373 leader_election.cc:288] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:03.707302 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:51 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:03.712455 13691 raft_consensus.cc:348] T 684caf5aaa8347e8b4e02323318fb0b7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:03.713769 13691 raft_consensus.cc:374] T 684caf5aaa8347e8b4e02323318fb0b7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:03.714017 13514 tablet_bootstrap.cc:492] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:33:03.714582 13514 ts_tablet_manager.cc:1397] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.262s user 0.006s sys 0.009s
I20260810 23:33:03.724900 13514 raft_consensus.cc:348] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:03.728616 13514 raft_consensus.cc:374] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:03.730034 13514 raft_consensus.cc:729] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:33:03.731876 13514 consensus_queue.cc:260] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:03.764930 13025 tablet_service.cc:1467] Processing CreateTablet for tablet afaab3b896674e18a222f999e3811d1d (DEFAULT_TABLE table=default.Table_b48833820bf944b688ae9585fb4daf68 [id=a6914cfeca1743cd84151fed87a4acee]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:33:03.776870 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 207478us
I20260810 23:33:03.777928 13691 raft_consensus.cc:729] T 684caf5aaa8347e8b4e02323318fb0b7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:03.780720 13014 tablet_service.cc:1467] Processing CreateTablet for tablet 182f4889e7484e989dec9bcbfc297bb9 (DEFAULT_TABLE table=default.Table_8371cab4a7ff44ceb3b02e4cb333fae1 [id=5f6c8ccff6664844be9afc52d4851118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:03.779140 13691 consensus_queue.cc:260] T 684caf5aaa8347e8b4e02323318fb0b7 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:33:03.783720 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 131977us
I20260810 23:33:03.785312 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]). Expected version 1 got 0
I20260810 23:33:03.794425 13010 tablet_service.cc:1467] Processing CreateTablet for tablet 4fd8ec477d17441ea018b5b957a36677 (DEFAULT_TABLE table=default.Table_88ec9ab54a584cd996eac5baed70b544 [id=9a3df87ccec0410faeee4aed54a34357]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:33:03.562958 12224 rpcz_store.cc:269] 0810 23:33:00.835099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.917273 (+ 82174us) service_pool.cc:227] Handling call
0810 23:33:00.941158 (+ 23885us) spinlock_profiling.cc:243] Waited 3.79 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:33:03.562549 (+2621391us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:02.622239 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:02.622717 (+ 478us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:02.622751 (+ 34us) write_op.cc:435] Acquired schema lock
0810 23:33:02.622768 (+ 17us) tablet.cc:660] Decoding operations
0810 23:33:02.623188 (+ 420us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:02.623245 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:02.623269 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:02.623582 (+ 313us) tablet.cc:699] Row locks acquired
0810 23:33:02.623599 (+ 17us) write_op.cc:260] PREPARE: finished
0810 23:33:02.623735 (+ 136us) write_op.cc:270] Start()
0810 23:33:02.623841 (+ 106us) write_op.cc:276] Timestamp: P: 1786404782623681 usec, L: 0
0810 23:33:02.623860 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 23:33:02.637751 (+ 13891us) log.cc:842] Serialized 919 byte log entry
0810 23:33:02.646104 (+ 8353us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:33:02.667088 (+ 20984us) op_driver.cc:444] REPLICATION: finished
0810 23:33:02.668302 (+ 1214us) write_op.cc:301] APPLY: starting
0810 23:33:02.668373 (+ 71us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:02.668501 (+ 128us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:02.668523 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:02.668883 (+ 360us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:02.669061 (+ 178us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:33:02.669106 (+ 45us) write_op.cc:312] APPLY: finished
0810 23:33:02.669346 (+ 240us) log.cc:842] Serialized 29 byte log entry
0810 23:33:02.670156 (+ 810us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:02.670247 (+ 91us) write_op.cc:454] Released schema lock
0810 23:33:02.670283 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:02.683289 (+ 13006us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:02.683323 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":18064,"reported_tablets":21,"spinlock_wait_cycles":8715648,"child_traces":[["op",{"apply.queue_time_us":1012,"apply.run_cpu_time_us":1954,"apply.run_wall_time_us":15286,"num_ops":2,"prepare.queue_time_us":187892,"prepare.run_cpu_time_us":4041,"prepare.run_wall_time_us":107223,"raft.queue_time_us":3364,"raft.run_cpu_time_us":960,"raft.run_wall_time_us":1104,"replication_time_us":43118,"spinlock_wait_cycles":30304128}]]}
I20260810 23:33:03.797602 13012 tablet_service.cc:1467] Processing CreateTablet for tablet 794688ecdc0945ed9014e2f747c7172c (DEFAULT_TABLE table=default.Table_9cab5b899a3748378d28d31c84e8b090 [id=b582cc38f8f04c8f989139034482272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:03.799425 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5]). Expected version 2 got 0
I20260810 23:33:03.804885 12237 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (table default.Table_4a7751c5b61c4332adeeb9ee00474b27 [id=2c8320255aab4036b5350d082f9e04ef]). Expected version 2 got 0
I20260810 23:33:03.809891 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 264) took 5937 ms. Trace:
I20260810 23:33:03.810425 14121 rpcz_store.cc:276] 0810 23:32:57.872567 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.876190 (+ 3623us) service_pool.cc:227] Handling call
0810 23:33:03.533591 (+5657401us) ts_tablet_manager.cc:1145] Deleting tablet 13ad5d63a2104995990a86a000963cdc
0810 23:33:03.809257 (+275666us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:03.809789 (+ 532us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5656642}
I20260810 23:33:03.811707 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 265) took 5934 ms. Trace:
I20260810 23:33:03.811791 12236 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_5a0a5857ffc649eb96a5e9e271b8ce7a" } new_table_name: "default.Table_dc485eb27eb64d70ba14404adc87fb9a" modify_external_catalogs: true
I20260810 23:33:03.812996 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afaab3b896674e18a222f999e3811d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.816694 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182f4889e7484e989dec9bcbfc297bb9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:03.820657 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 13ad5d63a2104995990a86a000963cdc because the tablet was not found. No further retry: Not found: Tablet not found: 13ad5d63a2104995990a86a000963cdc
I20260810 23:33:03.820844 15568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:03.821440 15568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:33:03.822161 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_523cb26bd2ea40f1b341060ff15317d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:03.824697 13010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fd8ec477d17441ea018b5b957a36677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.825481 15568 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:33:03.828975 13012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794688ecdc0945ed9014e2f747c7172c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.833949 14121 rpcz_store.cc:276] 0810 23:32:57.877445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.877571 (+ 126us) service_pool.cc:227] Handling call
0810 23:33:03.811367 (+5933796us) ts_tablet_manager.cc:1145] Deleting tablet db363eb0f98d460eb6f5a577233d8ac0
0810 23:33:03.811435 (+ 68us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:03.811647 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5932992}
I20260810 23:33:03.836176 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 266) took 5957 ms. Trace:
W20260810 23:33:03.823738 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_523cb26bd2ea40f1b341060ff15317d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:03.835832 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 427fa7e750e24ed9a48988fd3cb4ce1f (DEFAULT_TABLE table=default.Table_76308997ea5c4140ac84b26ccc7cc1bd [id=d90642b9034b48079d0c57fa25dd589e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:03.837033 14121 rpcz_store.cc:276] 0810 23:32:57.878835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.882707 (+ 3872us) service_pool.cc:227] Handling call
0810 23:33:03.835543 (+5952836us) ts_tablet_manager.cc:1145] Deleting tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea
0810 23:33:03.835642 (+ 99us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:03.836101 (+ 459us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5952056}
I20260810 23:33:03.838245 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427fa7e750e24ed9a48988fd3cb4ce1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.838333 14121 tablet_replica.cc:330] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:33:03.838605 12234 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (table default.Table_4a7751c5b61c4332adeeb9ee00474b27 [id=2c8320255aab4036b5350d082f9e04ef]). Expected version 2 got 0
I20260810 23:33:03.839367 14121 raft_consensus.cc:2229] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:03.839615 12234 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]). Expected version 1 got 0
I20260810 23:33:03.834597 13017 tablet_service.cc:1467] Processing CreateTablet for tablet b6884d44765c4640839a675a4a50c60e (DEFAULT_TABLE table=default.Table_566f96b5dead41b78406bb2776c7121b [id=350cca934f6840188a31c37cb981e8f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:03.743466 15082 raft_consensus.cc:2229] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:03.841017 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6884d44765c4640839a675a4a50c60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.841048 15082 raft_consensus.cc:2258] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:03.843731 15082 ts_tablet_manager.cc:1905] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:03.846900 12755 tablet_service.cc:1467] Processing CreateTablet for tablet 794688ecdc0945ed9014e2f747c7172c (DEFAULT_TABLE table=default.Table_9cab5b899a3748378d28d31c84e8b090 [id=b582cc38f8f04c8f989139034482272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:03.847198 12763 tablet_service.cc:1467] Processing CreateTablet for tablet b6884d44765c4640839a675a4a50c60e (DEFAULT_TABLE table=default.Table_566f96b5dead41b78406bb2776c7121b [id=350cca934f6840188a31c37cb981e8f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:03.848441 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794688ecdc0945ed9014e2f747c7172c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.849303 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 427fa7e750e24ed9a48988fd3cb4ce1f (DEFAULT_TABLE table=default.Table_76308997ea5c4140ac84b26ccc7cc1bd [id=d90642b9034b48079d0c57fa25dd589e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:33:03.750670 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 162827us
I20260810 23:33:03.853935 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427fa7e750e24ed9a48988fd3cb4ce1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.856837 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6884d44765c4640839a675a4a50c60e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:03.857069 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet db363eb0f98d460eb6f5a577233d8ac0 because the tablet was not found. No further retry: Not found: Tablet not found: db363eb0f98d460eb6f5a577233d8ac0
W20260810 23:33:03.858085 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d7f3b4e9ce34e0fb0fb7f68cb0fa1ea already in progress: creating tablet
I20260810 23:33:03.810742 12769 tablet_service.cc:1467] Processing CreateTablet for tablet afaab3b896674e18a222f999e3811d1d (DEFAULT_TABLE table=default.Table_b48833820bf944b688ae9585fb4daf68 [id=a6914cfeca1743cd84151fed87a4acee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:03.859771 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afaab3b896674e18a222f999e3811d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.813795 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 182f4889e7484e989dec9bcbfc297bb9 (DEFAULT_TABLE table=default.Table_8371cab4a7ff44ceb3b02e4cb333fae1 [id=5f6c8ccff6664844be9afc52d4851118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:03.814944 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 4fd8ec477d17441ea018b5b957a36677 (DEFAULT_TABLE table=default.Table_88ec9ab54a584cd996eac5baed70b544 [id=9a3df87ccec0410faeee4aed54a34357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:03.760766 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e8311c941b43058384cee6d8263dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:03.868479 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182f4889e7484e989dec9bcbfc297bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.873525 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fd8ec477d17441ea018b5b957a36677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:03.874147 13691 ts_tablet_manager.cc:1428] T 684caf5aaa8347e8b4e02323318fb0b7 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.358s user 0.008s sys 0.000s
I20260810 23:33:03.875221 13691 tablet_bootstrap.cc:492] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
W20260810 23:33:03.876967 12958 leader_election.cc:341] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: f47b8b57d5b44a018bc2d7b3f976d803
I20260810 23:33:03.878859 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684caf5aaa8347e8b4e02323318fb0b7" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:03.879487 13059 raft_consensus.cc:3037] T 684caf5aaa8347e8b4e02323318fb0b7 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:03.887214 13059 raft_consensus.cc:2454] T 684caf5aaa8347e8b4e02323318fb0b7 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:33:03.888583 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1452c292e9c476b9e26f56998f0ad55" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:03.889400 12806 raft_consensus.cc:2454] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:03.737279 13514 ts_tablet_manager.cc:1428] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260810 23:33:03.895613 13514 tablet_bootstrap.cc:492] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:33:03.899045 15570 tablet.cc:1722] T 046f1a7983ec40489edeaba4d2595b1c P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20260810 23:33:03.922533 14121 raft_consensus.cc:2258] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:03.925174 14121 ts_tablet_manager.cc:1905] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:03.928674 12240 catalog_manager.cc:5777] Not sending DeleteTablet requests; no consensus state for tablet 6ea17b2366bb47db9f6e49ca3052bdcd
I20260810 23:33:03.946460 12745 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:47536 (request call id 82) took 1281 ms. Trace:
I20260810 23:33:03.946949 12745 rpcz_store.cc:276] 0810 23:33:02.664291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:02.666193 (+ 1902us) service_pool.cc:227] Handling call
0810 23:33:03.243224 (+577031us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b5400160b54. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3b7268 00007f3cdf3a36cd 00007f3cddc4be04 00007f3cddc4bd92 00007f3cddc4bcc1 00007f3cddc4bc3c 00007f3cddc4aef2 00007f3cdc16c94c 00007f3cdc16bd86 00007f3cdc16b72f
0810 23:33:03.372917 (+129693us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54001220d4. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3b7268 00007f3cdf3a36cd 00007f3cddc4be04 00007f3cddc4bd92 00007f3cddc4bcc1 00007f3cddc4bc3c 00007f3cddc4aef2 00007f3cdc16c94c 00007f3cdc16bd86 00007f3cdc16b72f
0810 23:33:03.429419 (+ 56502us) spinlock_profiling.cc:243] Waited 54.8 ms on lock 0x7b5400140854. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3b7268 00007f3cdf3a36cd 00007f3cddc4be04 00007f3cddc4bd92 00007f3cddc4bcc1 00007f3cddc4bc3c 00007f3cddc4aef2 00007f3cdc16c94c 00007f3cdc16bd86 00007f3cdc16b72f
0810 23:33:03.748680 (+319261us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54001013d4. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3b7268 00007f3cdf3a36cd 00007f3cddc4be04 00007f3cddc4bd92 00007f3cddc4bcc1 00007f3cddc4bc3c 00007f3cddc4aef2 00007f3cdc16c94c 00007f3cdc16bd86 00007f3cdc16b72f
0810 23:33:03.754297 (+ 5617us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b5c000b050c. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde752361 00007f3cde8ebfa9 00007f3cde7de22a 00007f3cdf3a36a0 00007f3cddc4be04 00007f3cddc4bd92 00007f3cddc4bcc1 00007f3cddc4bc3c 00007f3cddc4aef2 00007f3cdc16c94c 00007f3cdc16bd86
0810 23:33:03.945489 (+191192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1664877568}
I20260810 23:33:03.952169 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
W20260810 23:33:03.963938 12229 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a72d6225d41e45ee86e00efd7270e44a consensus state has no leader
I20260810 23:33:03.996304 12223 master_service.cc:946] ReplaceTablet: received request to replace tablet 7031c257fdf34a28b6954d3d4827a8da from {username='slave'} at 127.0.0.1:34742
I20260810 23:33:03.997612 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684caf5aaa8347e8b4e02323318fb0b7" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:33:03.998283 13338 raft_consensus.cc:3037] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:03.999562 13665 tablet_bootstrap.cc:492] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:04.000005 13665 ts_tablet_manager.cc:1397] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.369s user 0.008s sys 0.011s
I20260810 23:33:04.002985 13665 raft_consensus.cc:348] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:04.003693 13665 raft_consensus.cc:374] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:04.007131 13338 raft_consensus.cc:2454] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:33:04.009722 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_4884ea66b39f44e99d957b62d1f38e0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:04.011480 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4884ea66b39f44e99d957b62d1f38e0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:04.014369 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_13d7888530b048f38e069ee640677b1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:04.022972 12226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 95622c5782e443b9903c85f64852eaa9
W20260810 23:33:04.025324 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13d7888530b048f38e069ee640677b1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:04.026381 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 95622c5782e443b9903c85f64852eaa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:03 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:04.028136 12495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:33:04.034945 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1452c292e9c476b9e26f56998f0ad55" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:04.031781 15210 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b; no voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:33:04.040356 15568 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260810 23:33:04.041226 12234 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
W20260810 23:33:04.044791 12181 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 62013.7us
I20260810 23:33:03.898945 15636 tablet.cc:1722] T 046f1a7983ec40489edeaba4d2595b1c P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20260810 23:33:04.050789 13282 tablet_service.cc:1467] Processing CreateTablet for tablet afaab3b896674e18a222f999e3811d1d (DEFAULT_TABLE table=default.Table_b48833820bf944b688ae9585fb4daf68 [id=a6914cfeca1743cd84151fed87a4acee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.052248 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afaab3b896674e18a222f999e3811d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:04.052660 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 95622c5782e443b9903c85f64852eaa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:03 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:04.056556 14121 ts_tablet_manager.cc:1918] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:04.060684 14121 log.cc:1196] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/c3d4c879ca6a402dbafe91940e7b367a
I20260810 23:33:04.061694 14121 ts_tablet_manager.cc:1939] T c3d4c879ca6a402dbafe91940e7b367a P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:33:04.069821 13280 tablet_service.cc:1467] Processing CreateTablet for tablet 182f4889e7484e989dec9bcbfc297bb9 (DEFAULT_TABLE table=default.Table_8371cab4a7ff44ceb3b02e4cb333fae1 [id=5f6c8ccff6664844be9afc52d4851118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.071746 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182f4889e7484e989dec9bcbfc297bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:04.075240 13213 leader_election.cc:302] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:33:04.076222 15617 raft_consensus.cc:2781] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:04.076814 15617 raft_consensus.cc:686] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:33:04.078291 15617 consensus_queue.cc:237] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:04.080884 13213 leader_election.cc:302] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:33:04.081800 15586 raft_consensus.cc:2781] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:04.082365 15586 raft_consensus.cc:482] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:04.082932 15586 raft_consensus.cc:3037] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:04.091223 13213 leader_election.cc:302] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:04.092507 15488 raft_consensus.cc:482] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:04.093479 15488 raft_consensus.cc:504] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:04.097195 12700 leader_election.cc:302] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:33:04.098093 15410 raft_consensus.cc:2781] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:04.098984 15410 raft_consensus.cc:686] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 LEADER]: Becoming Leader. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Running, Role: LEADER
I20260810 23:33:04.099709 15488 leader_election.cc:288] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:04.100015 15410 consensus_queue.cc:237] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:04.042164 12234 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 0d1b15dd545c4598aa5c7a9dc53618ec (table default.Table_0bf7fdea09a840b9a1394935cc53d643 [id=15edc7d801a94b40b8ab324e25d530e4]). Expected version 1 got 0
W20260810 23:33:04.073716 12236 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e17e5ec604044047b499312e27ca3de4 consensus state has no leader
I20260810 23:33:04.003942 13665 raft_consensus.cc:729] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:04.105363 13665 consensus_queue.cc:260] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:04.105942 12234 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5]). Expected version 2 got 0
I20260810 23:33:04.106503 15601 raft_consensus.cc:2726] T 684caf5aaa8347e8b4e02323318fb0b7 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:04.107478 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 267) took 6223 ms. Trace:
I20260810 23:33:04.107863 14121 rpcz_store.cc:276] 0810 23:32:57.884040 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:57.884308 (+ 268us) service_pool.cc:227] Handling call
0810 23:33:03.838123 (+5953815us) ts_tablet_manager.cc:1145] Deleting tablet c3d4c879ca6a402dbafe91940e7b367a
0810 23:33:03.838204 (+ 81us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:04.003138 (+164934us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.056274 (+ 53136us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.107405 (+ 51131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63009,"tablet-delete.queue_time_us":5952957}
I20260810 23:33:04.109637 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 794688ecdc0945ed9014e2f747c7172c (DEFAULT_TABLE table=default.Table_9cab5b899a3748378d28d31c84e8b090 [id=b582cc38f8f04c8f989139034482272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.109659 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 4fd8ec477d17441ea018b5b957a36677 (DEFAULT_TABLE table=default.Table_88ec9ab54a584cd996eac5baed70b544 [id=9a3df87ccec0410faeee4aed54a34357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.110327 14121 tablet_replica.cc:330] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:33:04.110935 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794688ecdc0945ed9014e2f747c7172c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:04.111182 14121 raft_consensus.cc:2229] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:04.111773 14121 raft_consensus.cc:2258] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:04.111951 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fd8ec477d17441ea018b5b957a36677. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:04.111963 12952 consensus_peers.cc:487] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:04.112861 12952 leader_election.cc:341] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: f47b8b57d5b44a018bc2d7b3f976d803
I20260810 23:33:04.113263 12952 leader_election.cc:302] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:04.114073 14121 ts_tablet_manager.cc:1905] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:04.114336 15628 raft_consensus.cc:2726] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:33:04.118891 13665 ts_tablet_manager.cc:1428] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.119s user 0.006s sys 0.000s
I20260810 23:33:04.120004 13665 tablet_bootstrap.cc:492] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:04.136116 15488 raft_consensus.cc:482] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:04.137018 15488 raft_consensus.cc:504] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:04.141110 15488 leader_election.cc:288] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:04.141381 13213 leader_election.cc:302] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:04.142467 15601 raft_consensus.cc:2781] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:04.143080 15601 raft_consensus.cc:482] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:04.143584 15601 raft_consensus.cc:3037] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:04.152551 15510 raft_consensus.cc:482] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:04.153580 15510 raft_consensus.cc:504] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:04.109583 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet c3d4c879ca6a402dbafe91940e7b367a (table default.Table_6e13e5c0a9da4cc5a1b85b4a247f8a3f [id=820d73489eed49d8a72357b3ac31300d]) successfully deleted
W20260810 23:33:04.157678 12215 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 48118.9us
I20260810 23:33:04.137925 12236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_3ae548119a244f178ea7e8222ab60395"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:04.139423 12226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_9e92a0778ad2454eb9ffe997c50710c9" } modify_external_catalogs: true
I20260810 23:33:04.159114 15510 leader_election.cc:288] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:04.159937 12226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_9e92a0778ad2454eb9ffe997c50710c9" } modify_external_catalogs: true
W20260810 23:33:04.166206 12236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ae548119a244f178ea7e8222ab60395 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:04.173175 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f68708a0b74a58b69581e8324a33f6" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:04.174233 12223 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_2c0b3093cb6e4b0f9e0fc90cff7fffbc" } new_table_name: "default.Table_91daf907141c4d738a6e3d655692427c" modify_external_catalogs: true
I20260810 23:33:04.182062 13277 tablet_service.cc:1467] Processing CreateTablet for tablet b6884d44765c4640839a675a4a50c60e (DEFAULT_TABLE table=default.Table_566f96b5dead41b78406bb2776c7121b [id=350cca934f6840188a31c37cb981e8f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.185645 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 427fa7e750e24ed9a48988fd3cb4ce1f (DEFAULT_TABLE table=default.Table_76308997ea5c4140ac84b26ccc7cc1bd [id=d90642b9034b48079d0c57fa25dd589e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.187970 13276 tablet_service.cc:1514] Processing DeleteTablet for tablet 95622c5782e443b9903c85f64852eaa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:03 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:04.206583 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc0e5cb08c44f7b9662d63905d99b02" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:04.222491 15272 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f26517150f1c480a877f15aec71d020b" instance_seqno: 1786404739485999) as {username='slave'} at 127.11.226.130:53995; Asking this server to re-register.
I20260810 23:33:04.227844 13025 tablet_service.cc:1467] Processing CreateTablet for tablet 0703e531759b4709bd3917cd9d307dd2 (DEFAULT_TABLE table=default.Table_95b3b58c2c4948c199a9b573367fa598 [id=56bede34f9a1429daaed5c6077be5513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.229367 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0703e531759b4709bd3917cd9d307dd2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:04.232898 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet e17e5ec604044047b499312e27ca3de4 (table default.Table_dc485eb27eb64d70ba14404adc87fb9a [id=b27a53844fa3419d80a7313ba2437c1e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e17e5ec604044047b499312e27ca3de4 consensus state has no leader
I20260810 23:33:04.248836 12235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_1476071cacef4a1b8681d049b2d23db5" } modify_external_catalogs: true
I20260810 23:33:04.249655 12235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_1476071cacef4a1b8681d049b2d23db5" } modify_external_catalogs: true
I20260810 23:33:04.260146 14121 ts_tablet_manager.cc:1918] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:33:04.260836 14121 log.cc:1196] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/f935119e4d9b4accb79db98185c80aca
I20260810 23:33:04.262104 14121 ts_tablet_manager.cc:1939] T f935119e4d9b4accb79db98185c80aca P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
W20260810 23:33:04.150875 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 53463.2us
I20260810 23:33:04.274616 13141 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:04.281996 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 273) took 4225 ms. Trace:
I20260810 23:33:04.287602 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 5135df28790d42d2be48053436c6cc60 (DEFAULT_TABLE table=default.Table_399593e632684b20a3032bb34a182dec [id=7184e7fca1b54e72a7ba15d5453baa37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.289091 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5135df28790d42d2be48053436c6cc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:04.211017 12756 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 322) took 1298 ms. Trace:
W20260810 23:33:04.217603 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 48214.7us
I20260810 23:33:04.231142 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 0703e531759b4709bd3917cd9d307dd2 (DEFAULT_TABLE table=default.Table_95b3b58c2c4948c199a9b573367fa598 [id=56bede34f9a1429daaed5c6077be5513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.294343 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0703e531759b4709bd3917cd9d307dd2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:04.305402 12698 leader_election.cc:341] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:04.308988 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 118647us
I20260810 23:33:04.289587 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc0e5cb08c44f7b9662d63905d99b02" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:04.292728 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f68708a0b74a58b69581e8324a33f6" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:04.315716 14121 rpcz_store.cc:276] 0810 23:33:00.056388 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.064131 (+ 7743us) service_pool.cc:227] Handling call
0810 23:33:04.110156 (+4046025us) ts_tablet_manager.cc:1145] Deleting tablet f935119e4d9b4accb79db98185c80aca
0810 23:33:04.110202 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:04.196525 (+ 86323us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.259708 (+ 63183us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.281917 (+ 22209us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3349,"tablet-delete.queue_time_us":4002191}
I20260810 23:33:04.318665 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet f935119e4d9b4accb79db98185c80aca (table default.Table_3e3ca6011776476baa1807ba610d0d3d [id=ad09aa494a7b4a3b8ade3a20807cbe0e]) successfully deleted
I20260810 23:33:04.325646 13514 tablet_bootstrap.cc:654] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:04.334681 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 5135df28790d42d2be48053436c6cc60 (DEFAULT_TABLE table=default.Table_399593e632684b20a3032bb34a182dec [id=7184e7fca1b54e72a7ba15d5453baa37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.297122 12756 rpcz_store.cc:276] 0810 23:33:02.912625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:02.912816 (+ 191us) service_pool.cc:227] Handling call
0810 23:33:03.282423 (+369607us) spinlock_profiling.cc:243] Waited 3.43 ms on lock 0x7b5800001898. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cd72acf11 00007f3cd7430851 00007f3cd7430702 00007f3ce02c7b09 00007f3cdf324730 00007f3cdf406d21 00007f3cdf3e41e3 00007f3cdf395869 00007f3cdda87c44 00007f3cdda87bd2 00007f3cdda87b01
0810 23:33:03.282556 (+ 133us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:33:03.282645 (+ 89us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:33:03.751382 (+468737us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.118706 (+367324us) tablet_replica.cc:205] Creating consensus instance
0810 23:33:04.210884 (+ 92178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7885568}
I20260810 23:33:04.299013 13691 tablet_bootstrap.cc:654] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:04.368126 12228 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_d3429684c4a7448caafcc4052426b361" } new_table_name: "default.Table_f1056e40bbd0468985b81347da8caef7" modify_external_catalogs: true
I20260810 23:33:04.374063 13277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6884d44765c4640839a675a4a50c60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:04.388458 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427fa7e750e24ed9a48988fd3cb4ce1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:04.411499 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 00b1e631390b4b69a6ba8c7b1ba5cfef (DEFAULT_TABLE table=default.Table_abf0a10256d44150b985a6d480568080 [id=ca29798e86b540b3aea30c9a32f4e4c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.420411 13023 tablet_service.cc:1467] Processing CreateTablet for tablet f60f59e756f94fd0b561ea7425a40c79 (DEFAULT_TABLE table=default.Table_86a07768a7834d14812dbdd318a66e5a [id=6bd548ee039f4ae89acbfbc935b09c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.424927 12770 tablet_service.cc:1467] Processing CreateTablet for tablet f60f59e756f94fd0b561ea7425a40c79 (DEFAULT_TABLE table=default.Table_86a07768a7834d14812dbdd318a66e5a [id=6bd548ee039f4ae89acbfbc935b09c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.426398 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60f59e756f94fd0b561ea7425a40c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:04.439136 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b1e631390b4b69a6ba8c7b1ba5cfef. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:04.323972 13212 leader_election.cc:341] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:04.440570 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 116654us
I20260810 23:33:04.450757 13691 tablet_bootstrap.cc:492] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:04.451422 13691 ts_tablet_manager.cc:1397] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.576s user 0.016s sys 0.004s
I20260810 23:33:04.453639 12229 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 0d1b15dd545c4598aa5c7a9dc53618ec (table default.Table_0bf7fdea09a840b9a1394935cc53d643 [id=15edc7d801a94b40b8ab324e25d530e4]). Expected version 1 got 0
I20260810 23:33:04.454566 12229 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_8a358dec4da84beda7fdf1eb485eb587 [id=b5cc65e503f7413fbade0c94e2e74b14]). Expected version 5 got 4
I20260810 23:33:04.463217 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_c608362e6ea74293b5e8a84421d98e10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:04.464290 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60f59e756f94fd0b561ea7425a40c79. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:04.465251 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c608362e6ea74293b5e8a84421d98e10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:04.469998 14121 tablet_replica.cc:330] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:33:04.470901 14121 raft_consensus.cc:2229] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:04.472538 14121 raft_consensus.cc:2258] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:04.443383 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc0e5cb08c44f7b9662d63905d99b02" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:04.474287 12806 raft_consensus.cc:2454] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:04.475507 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5135df28790d42d2be48053436c6cc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:04.476644 13691 raft_consensus.cc:348] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:04.485999 14121 ts_tablet_manager.cc:1905] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:04.499420 13691 raft_consensus.cc:374] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:04.500545 13691 raft_consensus.cc:729] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:04.506980 13691 consensus_queue.cc:260] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:04.512929 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 00b1e631390b4b69a6ba8c7b1ba5cfef (DEFAULT_TABLE table=default.Table_abf0a10256d44150b985a6d480568080 [id=ca29798e86b540b3aea30c9a32f4e4c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.514381 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b1e631390b4b69a6ba8c7b1ba5cfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:04.521198 15300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 9 FOLLOWER]: Advancing to term 10
I20260810 23:33:04.531611 13691 ts_tablet_manager.cc:1428] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.071s user 0.006s sys 0.002s
I20260810 23:33:04.532884 13691 tablet_bootstrap.cc:492] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:04.541461 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_7b45b2e7e9c242039e2f372c0491fdf7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:04.543524 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b45b2e7e9c242039e2f372c0491fdf7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:04.562990 12958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 51668.3us
W20260810 23:33:04.574374 12700 leader_election.cc:341] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:04.575212 12700 leader_election.cc:341] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:04.575639 12700 leader_election.cc:302] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:04.576463 15410 raft_consensus.cc:2726] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:04.568045 13691 tablet_bootstrap.cc:654] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:04.589601 12226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet d53029158b0142078d66bad842f0a69e
I20260810 23:33:04.592072 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet d53029158b0142078d66bad842f0a69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:04 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:04.592094 14121 ts_tablet_manager.cc:1918] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:04.592681 14121 log.cc:1196] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/4cdec13ee70d402fa3a08119e7bf1313
I20260810 23:33:04.593806 14121 ts_tablet_manager.cc:1939] T 4cdec13ee70d402fa3a08119e7bf1313 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:33:04.594589 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet d53029158b0142078d66bad842f0a69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:04 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:04.613339 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 275) took 4548 ms. Trace:
I20260810 23:33:04.617331 14121 rpcz_store.cc:276] 0810 23:33:00.064771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.065160 (+ 389us) service_pool.cc:227] Handling call
0810 23:33:04.317038 (+4251878us) ts_tablet_manager.cc:1145] Deleting tablet 4cdec13ee70d402fa3a08119e7bf1313
0810 23:33:04.469803 (+152765us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:04.528171 (+ 58368us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.587296 (+ 59125us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.613251 (+ 25955us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10512,"tablet-delete.queue_time_us":4208672}
I20260810 23:33:04.618927 15082 ts_tablet_manager.cc:1918] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:04.619370 15082 log.cc:1196] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/10033af79e854103bc47a046d2219ca0
I20260810 23:33:04.619575 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 4cdec13ee70d402fa3a08119e7bf1313 (table default.Table_1841b296cea84a33bb2cc47cfb03f460 [id=bc6adce8bdb44b54ae3021a1aaaa4fe7]) successfully deleted
I20260810 23:33:04.623222 13514 tablet_bootstrap.cc:492] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:33:04.623780 13514 ts_tablet_manager.cc:1397] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.728s user 0.015s sys 0.002s
I20260810 23:33:04.624686 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 277) took 4558 ms. Trace:
I20260810 23:33:04.626470 14121 rpcz_store.cc:276] 0810 23:33:00.066631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.067233 (+ 602us) service_pool.cc:227] Handling call
0810 23:33:04.624323 (+4557090us) ts_tablet_manager.cc:1145] Deleting tablet 9c29b11192604ac18c21b7bdc0238fb3
0810 23:33:04.624379 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:04.624617 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4515819}
I20260810 23:33:04.626791 13514 raft_consensus.cc:348] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:04.627521 13514 raft_consensus.cc:374] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:04.627774 13514 raft_consensus.cc:729] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
W20260810 23:33:04.628808 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 9c29b11192604ac18c21b7bdc0238fb3 because the tablet was not found. No further retry: Not found: Tablet not found: 9c29b11192604ac18c21b7bdc0238fb3
I20260810 23:33:04.633363 14121 tablet_replica.cc:330] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:33:04.637530 14121 raft_consensus.cc:2229] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:04.638285 14121 raft_consensus.cc:2258] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:04.646363 12236 master_service.cc:946] ReplaceTablet: received request to replace tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2 from {username='slave'} at 127.0.0.1:34742
I20260810 23:33:04.648571 14121 ts_tablet_manager.cc:1905] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:33:04.650344 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 75142.7us
I20260810 23:33:04.652894 15628 raft_consensus.cc:482] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f54aa78a60784a3bb2f5eb4bb2401ce3)
I20260810 23:33:04.674074 15628 raft_consensus.cc:504] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:04.676420 15628 leader_election.cc:288] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:33:04.665468 13018 tablet_service.cc:1467] Processing CreateTablet for tablet d491aa5218ca4fb2a496d319a0cac592 (DEFAULT_TABLE table=default.Table_496ebacf57f14572b620d213d8865579 [id=0e6874a7b48d40dd830fcdb02378f973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:04.677637 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d491aa5218ca4fb2a496d319a0cac592. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:04.695446 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c53785fdd24df79f138df6176e75d2" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:04.698117 15410 raft_consensus.cc:482] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:04.699934 15410 raft_consensus.cc:504] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:04.677340 15082 ts_tablet_manager.cc:1939] T 10033af79e854103bc47a046d2219ca0 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:33:04.706171 15410 leader_election.cc:288] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:04.712420 15636 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:47546 (request call id 320) took 1847 ms. Trace:
I20260810 23:33:04.513079 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc0e5cb08c44f7b9662d63905d99b02" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:04.723582 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975a9bab969e46d2b939fc76ce350b99" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:04.693718 15486 ts_tablet_manager.cc:1918] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:33:04.727788 12768 tablet_service.cc:1467] Processing CreateTablet for tablet d491aa5218ca4fb2a496d319a0cac592 (DEFAULT_TABLE table=default.Table_496ebacf57f14572b620d213d8865579 [id=0e6874a7b48d40dd830fcdb02378f973]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:33:04.729282 12952 leader_election.cc:341] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 88c53785fdd24df79f138df6176e75d2
I20260810 23:33:04.729596 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d491aa5218ca4fb2a496d319a0cac592. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:04.729724 12952 leader_election.cc:302] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:04.730944 15628 raft_consensus.cc:2726] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:33:04.733134 15486 log.cc:1196] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/ab8d987109ba40198645881b449cb972
I20260810 23:33:04.759531 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 288) took 5139 ms. Trace:
I20260810 23:33:04.760424 15082 rpcz_store.cc:276] 0810 23:32:59.620584 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:59.620751 (+ 167us) service_pool.cc:227] Handling call
0810 23:33:03.524433 (+3903682us) ts_tablet_manager.cc:1145] Deleting tablet 10033af79e854103bc47a046d2219ca0
0810 23:33:03.524479 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:03.874915 (+350436us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.618636 (+743721us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.759426 (+140790us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":212502,"tablet-delete.queue_time_us":3902787}
I20260810 23:33:04.728142 13334 raft_consensus.cc:2454] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:33:04.761953 15656 tablet.cc:1722] T 046f1a7983ec40489edeaba4d2595b1c P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20260810 23:33:04.777045 12223 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba8d56858471421a9fbc361ee6aff28f consensus state has no leader
I20260810 23:33:04.779417 12223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_c28d62ee4c2a432b82a28cc0111c4e4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:04.782692 12223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c28d62ee4c2a432b82a28cc0111c4e4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:04.751106 15586 consensus_meta.cc:287] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent flushing consensus metadata: real 0.665s user 0.003s sys 0.010s
I20260810 23:33:04.794379 15586 raft_consensus.cc:504] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:04.798497 15586 leader_election.cc:288] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:33:04.799994 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 398588us
I20260810 23:33:04.802690 13338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975a9bab969e46d2b939fc76ce350b99" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:04.814209 14121 ts_tablet_manager.cc:1918] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:04.815521 14121 log.cc:1196] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/f47b8b57d5b44a018bc2d7b3f976d803
I20260810 23:33:04.817005 13409 heartbeater.cc:347] Connected to a master server at 127.11.226.190:43533
I20260810 23:33:04.817564 14121 ts_tablet_manager.cc:1939] T f47b8b57d5b44a018bc2d7b3f976d803 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:33:04.789084 15486 ts_tablet_manager.cc:1939] T ab8d987109ba40198645881b449cb972 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:04.840226 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 280) took 4772 ms. Trace:
I20260810 23:33:04.850620 14121 rpcz_store.cc:276] 0810 23:33:00.067948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.068207 (+ 259us) service_pool.cc:227] Handling call
0810 23:33:04.633169 (+4564962us) ts_tablet_manager.cc:1145] Deleting tablet f47b8b57d5b44a018bc2d7b3f976d803
0810 23:33:04.633221 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:04.722805 (+ 89584us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.813761 (+ 90956us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.831876 (+ 18115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4524455}
I20260810 23:33:04.845934 13213 leader_election.cc:302] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:04.853041 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet f47b8b57d5b44a018bc2d7b3f976d803 (table default.Table_7ba7fd44e98542e78c604739a5086a91 [id=a7a55f9e68f04a9c9c43b9607cd901c6]) successfully deleted
I20260810 23:33:04.855113 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 281) took 4785 ms. Trace:
I20260810 23:33:04.858182 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 317) took 1440 ms. Trace:
I20260810 23:33:04.858789 14121 rpcz_store.cc:276] 0810 23:33:00.069314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.069537 (+ 223us) service_pool.cc:227] Handling call
0810 23:33:04.854630 (+4785093us) ts_tablet_manager.cc:1145] Deleting tablet fd9d90e1ceab443dbec12b2c81b975e8
0810 23:33:04.854702 (+ 72us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:04.855023 (+ 321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4745565}
I20260810 23:33:04.858640 15486 rpcz_store.cc:276] 0810 23:33:03.417343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.417621 (+ 278us) service_pool.cc:227] Handling call
0810 23:33:03.624702 (+207081us) ts_tablet_manager.cc:1145] Deleting tablet ab8d987109ba40198645881b449cb972
0810 23:33:03.624803 (+ 101us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:03.912787 (+287984us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.509909 (+597122us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.858076 (+348167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":195}
W20260810 23:33:04.861013 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet fd9d90e1ceab443dbec12b2c81b975e8 because the tablet was not found. No further retry: Not found: Tablet not found: fd9d90e1ceab443dbec12b2c81b975e8
I20260810 23:33:04.863369 14121 tablet_replica.cc:330] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:33:04.868908 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 320) took 1225 ms. Trace:
I20260810 23:33:04.869416 15486 rpcz_store.cc:276] 0810 23:33:03.643661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.643870 (+ 209us) service_pool.cc:227] Handling call
0810 23:33:04.868467 (+1224597us) ts_tablet_manager.cc:1145] Deleting tablet 4cdec13ee70d402fa3a08119e7bf1313
0810 23:33:04.868538 (+ 71us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:04.868813 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1223939}
I20260810 23:33:04.866608 14121 raft_consensus.cc:2229] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:04.871099 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 322) took 1206 ms. Trace:
I20260810 23:33:04.873984 14121 pending_rounds.cc:62] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b: Trying to abort 1 pending ops.
W20260810 23:33:04.877838 15601 consensus_meta.cc:287] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent flushing consensus metadata: real 0.734s user 0.009s sys 0.009s
I20260810 23:33:04.881137 14121 pending_rounds.cc:68] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317113964290306048 op_type: NO_OP noop_request { }
I20260810 23:33:04.881757 13665 tablet_bootstrap.cc:654] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:04.884132 14121 raft_consensus.cc:2866] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:33:04.884773 14121 raft_consensus.cc:2258] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:04.891374 15461 raft_consensus.cc:2781] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:04.891795 15461 raft_consensus.cc:482] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:04.892134 15461 raft_consensus.cc:3037] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:04.903368 14121 ts_tablet_manager.cc:1905] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:33:04.912763 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 471303us
I20260810 23:33:04.713083 15636 rpcz_store.cc:276] 0810 23:33:02.864889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:02.888210 (+ 23321us) service_pool.cc:227] Handling call
0810 23:33:04.712302 (+1824092us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:03.142904 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 23:33:03.143325 (+ 421us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 23:33:03.143359 (+ 34us) alter_schema_op.cc:65] Acquired schema lock
0810 23:33:03.143420 (+ 61us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 23:33:03.143546 (+ 126us) alter_schema_op.cc:126] START. Timestamp: 1786404783143754
0810 23:33:03.143566 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:33:03.144286 (+ 720us) log.cc:842] Serialized 173 byte log entry
0810 23:33:03.855626 (+711340us) op_driver.cc:444] REPLICATION: finished
0810 23:33:03.898794 (+ 43168us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 23:33:04.047014 (+148220us) log.cc:842] Serialized 5 byte log entry
0810 23:33:04.505756 (+458742us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:04.699969 (+194213us) log.cc:842] Serialized 14 byte log entry
0810 23:33:04.701948 (+ 1979us) alter_schema_op.cc:71] Released schema lock
0810 23:33:04.702007 (+ 59us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":42934,"prepare.queue_time_us":253639,"prepare.run_cpu_time_us":10877,"prepare.run_wall_time_us":341278,"replication_time_us":711946,"thread_start_us":612904,"threads_started":3,"wal-append.queue_time_us":331304}]]}
W20260810 23:33:04.940968 13212 leader_election.cc:341] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:04.954614 15672 tablet.cc:1722] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:33:04.871536 15486 rpcz_store.cc:276] 0810 23:33:03.664464 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.664695 (+ 231us) service_pool.cc:227] Handling call
0810 23:33:04.870707 (+1206012us) ts_tablet_manager.cc:1145] Deleting tablet f47b8b57d5b44a018bc2d7b3f976d803
0810 23:33:04.870768 (+ 61us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:04.871025 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1200294}
I20260810 23:33:04.984934 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 10033af79e854103bc47a046d2219ca0 (table default.Table_620c572fc75b44fe8ca892789875c1f7 [id=cf58351bb8594bdbb92168f093ec91a0]) successfully deleted
I20260810 23:33:04.878398 15601 raft_consensus.cc:504] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:04.995345 15601 leader_election.cc:288] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:05.005800 14121 ts_tablet_manager.cc:1918] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:05.006235 14121 log.cc:1196] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/4d0d9227d8574de8bd73041f3e8043a0
I20260810 23:33:05.006982 14121 ts_tablet_manager.cc:1939] T 4d0d9227d8574de8bd73041f3e8043a0 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:33:04.974489 15676 raft_consensus.cc:482] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:05.022717 15676 raft_consensus.cc:504] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:04.981232 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 292) took 4684 ms. Trace:
I20260810 23:33:05.024365 15082 rpcz_store.cc:276] 0810 23:33:00.296642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.296887 (+ 245us) service_pool.cc:227] Handling call
0810 23:33:04.980604 (+4683717us) ts_tablet_manager.cc:1145] Deleting tablet 422b25457d1549e186641fcedd8feab9
0810 23:33:04.980674 (+ 70us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:04.981114 (+ 440us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4682604}
I20260810 23:33:05.025321 15676 leader_election.cc:288] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:05.026190 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 293) took 4037 ms. Trace:
I20260810 23:33:05.060959 15082 rpcz_store.cc:276] 0810 23:33:00.988772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.988959 (+ 187us) service_pool.cc:227] Handling call
0810 23:33:05.025816 (+4036857us) ts_tablet_manager.cc:1145] Deleting tablet f47b8b57d5b44a018bc2d7b3f976d803
0810 23:33:05.025893 (+ 77us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:05.026129 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4036099}
W20260810 23:33:04.986559 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 176573us
I20260810 23:33:04.926628 13514 consensus_queue.cc:260] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:33:04.993700 12698 leader_election.cc:341] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:05.062850 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 69202.8us
W20260810 23:33:05.064781 12700 leader_election.cc:341] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:05.065572 12698 leader_election.cc:302] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:33:05.066896 12700 leader_election.cc:302] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:05.068106 15676 raft_consensus.cc:2781] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:05.068539 15676 raft_consensus.cc:482] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:05.068740 15410 raft_consensus.cc:2726] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:05.068991 15676 raft_consensus.cc:3037] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:05.075757 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806fe5162aef4d36ad5e2ac324047426" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:05.077171 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a480093c481941f0bb35949c7606f37f" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:05.078161 12803 raft_consensus.cc:3037] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:05.092690 15082 tablet_replica.cc:330] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
W20260810 23:33:05.093740 12217 catalog_manager.cc:4808] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet f47b8b57d5b44a018bc2d7b3f976d803 because the tablet was not found. No further retry: Not found: Tablet not found: f47b8b57d5b44a018bc2d7b3f976d803
I20260810 23:33:05.103607 13691 tablet_bootstrap.cc:492] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
W20260810 23:33:05.083422 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 120489us
I20260810 23:33:05.124500 13691 ts_tablet_manager.cc:1397] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.592s user 0.017s sys 0.005s
I20260810 23:33:05.128821 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 276) took 5062 ms. Trace:
I20260810 23:33:05.131796 14121 rpcz_store.cc:276] 0810 23:33:00.066180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.066408 (+ 228us) service_pool.cc:227] Handling call
0810 23:33:04.863120 (+4796712us) ts_tablet_manager.cc:1145] Deleting tablet 4d0d9227d8574de8bd73041f3e8043a0
0810 23:33:04.863187 (+ 67us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:04.986120 (+122933us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:05.005544 (+ 19424us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:05.128727 (+123183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":866,"tablet-delete.queue_time_us":4754729}
I20260810 23:33:05.133495 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 4d0d9227d8574de8bd73041f3e8043a0 (table default.Table_bf4e391ee20b45d393a269203b401bc0 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]) successfully deleted
I20260810 23:33:05.134318 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 279) took 5066 ms. Trace:
I20260810 23:33:05.134690 14121 rpcz_store.cc:276] 0810 23:33:00.067516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.080377 (+ 12861us) service_pool.cc:227] Handling call
0810 23:33:05.133959 (+5053582us) ts_tablet_manager.cc:1145] Deleting tablet 423e8d583d704fa9834026ebcd57e17e
0810 23:33:05.134016 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:05.134252 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5023948}
W20260810 23:33:05.136161 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 423e8d583d704fa9834026ebcd57e17e because the tablet was not found. No further retry: Not found: Tablet not found: 423e8d583d704fa9834026ebcd57e17e
I20260810 23:33:05.136533 14121 tablet_replica.cc:330] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:33:05.137650 14121 raft_consensus.cc:2229] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:05.138859 14121 raft_consensus.cc:2258] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:05.141475 14121 ts_tablet_manager.cc:1905] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:05.096180 13514 ts_tablet_manager.cc:1428] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.472s user 0.007s sys 0.001s
W20260810 23:33:05.151757 13514 ts_tablet_manager.cc:1460] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3: Tablet startup took 527ms
W20260810 23:33:05.152123 13514 ts_tablet_manager.cc:1462] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3: Trace:
0810 23:32:46.032858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:32:46.033105 (+ 247us) service_pool.cc:227] Handling call
0810 23:32:46.049866 (+ 16761us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:32:46.050820 (+ 954us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:32:46.128634 (+ 77814us) tablet_metadata.cc:641] Metadata flushed
0810 23:32:46.145624 (+ 16990us) tablet_replica.cc:205] Creating consensus instance
0810 23:32:46.146692 (+ 1068us) inbound_call.cc:173] Queueing success response
0810 23:33:03.895204 (+17748512us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0810 23:33:04.624228 (+729024us) ts_tablet_manager.cc:1429] Starting tablet replica
0810 23:33:04.624876 (+ 648us) tablet_replica.cc:259] Starting instrumentation
0810 23:33:04.625515 (+ 639us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":2235,"raft.run_cpu_time_us":549,"raft.run_wall_time_us":761,"tablet-open.queue_time_us":17748919}
I20260810 23:33:05.158399 15410 raft_consensus.cc:482] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f54aa78a60784a3bb2f5eb4bb2401ce3)
I20260810 23:33:05.161351 15410 raft_consensus.cc:504] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:05.164719 15410 leader_election.cc:288] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:33:05.172251 13022 consensus_meta.cc:287] T 00b1e631390b4b69a6ba8c7b1ba5cfef P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.709s user 0.010s sys 0.002s
I20260810 23:33:05.186602 15410 raft_consensus.cc:482] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:05.187776 15410 raft_consensus.cc:504] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:05.191533 15410 leader_election.cc:288] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:05.193780 13691 raft_consensus.cc:348] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:05.194754 13691 raft_consensus.cc:374] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:05.195119 13691 raft_consensus.cc:729] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:05.211961 15486 tablet_replica.cc:330] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:05.213136 15486 raft_consensus.cc:2229] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:05.213290 13334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b25457d1549e186641fcedd8feab9" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:05.213887 15486 raft_consensus.cc:2258] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:05.218577 13277 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 334) took 1038 ms. Trace:
I20260810 23:33:05.218963 13277 rpcz_store.cc:276] 0810 23:33:04.180116 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.180330 (+ 214us) service_pool.cc:227] Handling call
0810 23:33:04.183288 (+ 2958us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:33:04.183452 (+ 164us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:33:04.774026 (+590574us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:05.218039 (+444013us) tablet_replica.cc:205] Creating consensus instance
0810 23:33:05.218506 (+ 467us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:33:05.236366 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806fe5162aef4d36ad5e2ac324047426" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:33:05.240609 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet e17e5ec604044047b499312e27ca3de4 (table default.Table_dc485eb27eb64d70ba14404adc87fb9a [id=b27a53844fa3419d80a7313ba2437c1e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e17e5ec604044047b499312e27ca3de4 consensus state has no leader
I20260810 23:33:05.242498 15461 raft_consensus.cc:504] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:05.247676 15461 leader_election.cc:288] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:05.292101 15486 ts_tablet_manager.cc:1905] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:05.254662 13691 consensus_queue.cc:260] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:33:05.262687 15300 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.127s sys 0.145s
W20260810 23:33:05.332278 15300 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.659s user 0.128s sys 0.147s
I20260810 23:33:05.333323 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c53785fdd24df79f138df6176e75d2" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:05.334121 13057 raft_consensus.cc:2454] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 1.
I20260810 23:33:05.334049 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b25457d1549e186641fcedd8feab9" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:33:05.206542 13018 consensus_meta.cc:287] T d491aa5218ca4fb2a496d319a0cac592 P f26517150f1c480a877f15aec71d020b: Time spent flushing consensus metadata: real 0.516s user 0.009s sys 0.000s
I20260810 23:33:05.252377 15677 tablet.cc:1722] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:33:05.198971 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1452c292e9c476b9e26f56998f0ad55" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:05.297776 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_5b5e98d980b14439a4826410a95f81cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:05.339485 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b5e98d980b14439a4826410a95f81cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:05.340265 13665 tablet_bootstrap.cc:492] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:05.295739 12226 catalog_manager.cc:5495] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129). New cstate: current_term: 1 leader_uuid: "191d120f4c7f44fc8475305a6d2e75f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: HEALTHY } } }
I20260810 23:33:05.340828 13665 ts_tablet_manager.cc:1397] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 1.221s user 0.012s sys 0.010s
I20260810 23:33:05.341205 12226 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:33:05.303133 13280 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 331) took 1234 ms. Trace:
I20260810 23:33:05.344187 12226 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]). Expected version 1 got 0
I20260810 23:33:05.345427 12226 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 00f68708a0b74a58b69581e8324a33f6 (table default.Table_d134a55028664442b08b229593877973 [id=26be70827ca442a4b24a26015ffd952a]). Expected version 2 got 0
I20260810 23:33:05.346127 12226 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (table default.Table_4a7751c5b61c4332adeeb9ee00474b27 [id=2c8320255aab4036b5350d082f9e04ef]). Expected version 2 got 0
W20260810 23:33:05.301610 12228 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e27986aa12e44ace92b335421a457348 consensus state has no leader
I20260810 23:33:05.325172 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet ab8d987109ba40198645881b449cb972 (table default.Table_d8a0e3e655b9417d9df3ff6d82b76f15 [id=d0b86eafe9544cd98d8fce8ece3b5c10]) successfully deleted
W20260810 23:33:05.347756 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 4cdec13ee70d402fa3a08119e7bf1313 because the tablet was not found. No further retry: Not found: Tablet not found: 4cdec13ee70d402fa3a08119e7bf1313
I20260810 23:33:05.201150 13514 tablet_bootstrap.cc:492] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:33:05.203173 15636 tablet.cc:1722] T 8fb2ee1c9b41478da001aa67d7a7ad3c P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:33:05.348785 12228 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 387) took 1008 ms. Trace:
I20260810 23:33:05.233687 15676 raft_consensus.cc:504] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:33:05.235008 12698 leader_election.cc:341] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 88c53785fdd24df79f138df6176e75d2
W20260810 23:33:05.265616 12770 consensus_meta.cc:287] T f60f59e756f94fd0b561ea7425a40c79 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.594s user 0.012s sys 0.000s
I20260810 23:33:05.301375 15668 raft_consensus.cc:482] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:33:05.355347 13141 heartbeater.cc:415] Failed 1 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:33:05.355945 13141 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:05.343683 13280 rpcz_store.cc:276] 0810 23:33:04.068548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.068697 (+ 149us) service_pool.cc:227] Handling call
0810 23:33:04.070999 (+ 2302us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:33:04.071120 (+ 121us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:33:05.045602 (+974482us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:05.223991 (+178389us) tablet_replica.cc:205] Creating consensus instance
0810 23:33:05.303043 (+ 79052us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:33:05.271886 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a480093c481941f0bb35949c7606f37f" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:05.327457 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1452c292e9c476b9e26f56998f0ad55" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:05.360854 13691 ts_tablet_manager.cc:1428] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.215s user 0.008s sys 0.000s
I20260810 23:33:05.349114 12228 rpcz_store.cc:276] 0810 23:33:04.340254 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.367439 (+ 27185us) service_pool.cc:227] Handling call
0810 23:33:04.368912 (+ 1473us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:33:04.369709 (+ 797us) catalog_manager.cc:3616] Apply alter schema
0810 23:33:04.370424 (+ 715us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:33:04.370687 (+ 263us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:33:05.119950 (+749263us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:33:05.348710 (+228760us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:04.997431 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:04.997863 (+ 432us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:04.997903 (+ 40us) write_op.cc:435] Acquired schema lock
0810 23:33:04.997917 (+ 14us) tablet.cc:660] Decoding operations
0810 23:33:04.998272 (+ 355us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:04.998347 (+ 75us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:04.998384 (+ 37us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:04.998612 (+ 228us) tablet.cc:699] Row locks acquired
0810 23:33:04.998633 (+ 21us) write_op.cc:260] PREPARE: finished
0810 23:33:04.998783 (+ 150us) write_op.cc:270] Start()
0810 23:33:04.998922 (+ 139us) write_op.cc:276] Timestamp: P: 1786404784998707 usec, L: 0
0810 23:33:04.998960 (+ 38us) op_driver.cc:331] REPLICATION: starting
0810 23:33:04.999724 (+ 764us) log.cc:842] Serialized 405 byte log entry
0810 23:33:05.103200 (+103476us) op_driver.cc:444] REPLICATION: finished
0810 23:33:05.116806 (+ 13606us) write_op.cc:301] APPLY: starting
0810 23:33:05.116941 (+ 135us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:05.117119 (+ 178us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:05.117275 (+ 156us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:05.117749 (+ 474us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:05.118017 (+ 268us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:05.118100 (+ 83us) write_op.cc:312] APPLY: finished
0810 23:33:05.118476 (+ 376us) log.cc:842] Serialized 23 byte log entry
0810 23:33:05.119421 (+ 945us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:05.119524 (+ 103us) write_op.cc:454] Released schema lock
0810 23:33:05.119596 (+ 72us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:05.120572 (+ 976us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:05.120682 (+ 110us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":13228,"apply.run_cpu_time_us":3202,"apply.run_wall_time_us":8184,"mutex_wait_us":38138,"num_ops":1,"prepare.queue_time_us":624926,"prepare.run_cpu_time_us":2732,"prepare.run_wall_time_us":57969,"replication_time_us":104122}]]}
W20260810 23:33:05.367281 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet f47b8b57d5b44a018bc2d7b3f976d803 because the tablet was not found. No further retry: Not found: Tablet not found: f47b8b57d5b44a018bc2d7b3f976d803
I20260810 23:33:05.374084 13665 raft_consensus.cc:348] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:05.374840 12235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 8fb2ee1c9b41478da001aa67d7a7ad3c
I20260810 23:33:05.374927 13665 raft_consensus.cc:374] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:05.375166 13665 raft_consensus.cc:729] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:05.375779 13665 consensus_queue.cc:260] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:05.378159 12235 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 386) took 1131 ms. Trace:
I20260810 23:33:05.378495 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:04 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:05.381359 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 00b1e631390b4b69a6ba8c7b1ba5cfef (DEFAULT_TABLE table=default.Table_abf0a10256d44150b985a6d480568080 [id=ca29798e86b540b3aea30c9a32f4e4c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:05.382748 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b1e631390b4b69a6ba8c7b1ba5cfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:05.385787 13691 tablet_bootstrap.cc:492] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:05.378592 12235 rpcz_store.cc:276] 0810 23:33:04.246752 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.248213 (+ 1461us) service_pool.cc:227] Handling call
0810 23:33:04.250129 (+ 1916us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:33:04.250752 (+ 623us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:33:04.250925 (+ 173us) catalog_manager.cc:2761] Locking tablets
0810 23:33:04.251545 (+ 620us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:33:05.054743 (+803198us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:33:05.302056 (+247313us) catalog_manager.cc:2809] Committing in-memory state
0810 23:33:05.374696 (+ 72640us) catalog_manager.cc:2813] Aborting table tasks
0810 23:33:05.378083 (+ 3387us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:04.810454 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:04.811023 (+ 569us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:04.811061 (+ 38us) write_op.cc:435] Acquired schema lock
0810 23:33:04.811085 (+ 24us) tablet.cc:660] Decoding operations
0810 23:33:04.811567 (+ 482us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:04.811612 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:04.811647 (+ 35us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:04.812093 (+ 446us) tablet.cc:699] Row locks acquired
0810 23:33:04.812110 (+ 17us) write_op.cc:260] PREPARE: finished
0810 23:33:04.947063 (+134953us) write_op.cc:270] Start()
0810 23:33:04.947206 (+ 143us) write_op.cc:276] Timestamp: P: 1786404784946974 usec, L: 0
0810 23:33:04.947223 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:33:04.994920 (+ 47697us) log.cc:842] Serialized 799 byte log entry
0810 23:33:05.049623 (+ 54703us) op_driver.cc:444] REPLICATION: finished
0810 23:33:05.051320 (+ 1697us) write_op.cc:301] APPLY: starting
0810 23:33:05.051505 (+ 185us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:05.051717 (+ 212us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:05.051983 (+ 266us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:05.052651 (+ 668us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:05.052944 (+ 293us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:33:05.053036 (+ 92us) write_op.cc:312] APPLY: finished
0810 23:33:05.053390 (+ 354us) log.cc:842] Serialized 29 byte log entry
0810 23:33:05.054293 (+ 903us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:05.054424 (+ 131us) write_op.cc:454] Released schema lock
0810 23:33:05.054481 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:05.054708 (+ 227us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:05.054778 (+ 70us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1360,"apply.run_cpu_time_us":3059,"apply.run_wall_time_us":3804,"num_ops":2,"prepare.queue_time_us":548512,"prepare.run_cpu_time_us":3294,"prepare.run_wall_time_us":186826,"replication_time_us":102256}]]}
I20260810 23:33:05.389034 13280 tablet_service.cc:1467] Processing CreateTablet for tablet 0703e531759b4709bd3917cd9d307dd2 (DEFAULT_TABLE table=default.Table_95b3b58c2c4948c199a9b573367fa598 [id=56bede34f9a1429daaed5c6077be5513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:05.390354 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0703e531759b4709bd3917cd9d307dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:05.393090 13277 tablet_service.cc:1467] Processing CreateTablet for tablet 5135df28790d42d2be48053436c6cc60 (DEFAULT_TABLE table=default.Table_399593e632684b20a3032bb34a182dec [id=7184e7fca1b54e72a7ba15d5453baa37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:05.394351 13277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5135df28790d42d2be48053436c6cc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:05.397213 13282 tablet_service.cc:1467] Processing CreateTablet for tablet f60f59e756f94fd0b561ea7425a40c79 (DEFAULT_TABLE table=default.Table_86a07768a7834d14812dbdd318a66e5a [id=6bd548ee039f4ae89acbfbc935b09c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:05.398430 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60f59e756f94fd0b561ea7425a40c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:05.408316 15672 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:53280 (request call id 329) took 1505 ms. Trace:
I20260810 23:33:05.410583 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet d53029158b0142078d66bad842f0a69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:04 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:05.416479 13290 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53280 (request call id 335) took 1232 ms. Trace:
I20260810 23:33:05.420058 12236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2
I20260810 23:33:05.421995 12236 catalog_manager.cc:6565] ReplaceTablet: tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2 deleted and replaced by tablet 403caf051714438bb77fc2fbefca15c5
I20260810 23:33:05.425230 12763 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 403caf051714438bb77fc2fbefca15c5) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:05.427417 12236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_94e24b4b27d047a99df427c453fe1752" } modify_external_catalogs: true
I20260810 23:33:05.428224 12236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_94e24b4b27d047a99df427c453fe1752" } modify_external_catalogs: true
I20260810 23:33:05.433151 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 403caf051714438bb77fc2fbefca15c5) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:05.408851 15672 rpcz_store.cc:276] 0810 23:33:03.903120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.903247 (+ 127us) service_pool.cc:227] Handling call
0810 23:33:05.408193 (+1504946us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:03.946368 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 23:33:03.947109 (+ 741us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 23:33:03.947176 (+ 67us) alter_schema_op.cc:65] Acquired schema lock
0810 23:33:03.947356 (+ 180us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 23:33:03.947628 (+ 272us) alter_schema_op.cc:126] START. Timestamp: 1786404783945368
0810 23:33:03.947681 (+ 53us) op_driver.cc:331] REPLICATION: starting
0810 23:33:04.038365 (+ 90684us) log.cc:842] Serialized 173 byte log entry
0810 23:33:04.902053 (+863688us) op_driver.cc:444] REPLICATION: finished
0810 23:33:04.954373 (+ 52320us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 23:33:04.981282 (+ 26909us) log.cc:842] Serialized 5 byte log entry
0810 23:33:05.376817 (+395535us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:05.407628 (+ 30811us) log.cc:842] Serialized 14 byte log entry
0810 23:33:05.407767 (+ 139us) alter_schema_op.cc:71] Released schema lock
0810 23:33:05.407897 (+ 130us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52059,"prepare.queue_time_us":41943,"prepare.run_cpu_time_us":2497,"prepare.run_wall_time_us":96610,"replication_time_us":954255,"thread_start_us":145093,"threads_started":2}]]}
I20260810 23:33:05.457639 13291 tablet_service.cc:1467] Processing CreateTablet for tablet d491aa5218ca4fb2a496d319a0cac592 (DEFAULT_TABLE table=default.Table_496ebacf57f14572b620d213d8865579 [id=0e6874a7b48d40dd830fcdb02378f973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:05.458799 13665 ts_tablet_manager.cc:1428] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.118s user 0.007s sys 0.000s
I20260810 23:33:05.416914 13290 rpcz_store.cc:276] 0810 23:33:04.184519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.184660 (+ 141us) service_pool.cc:227] Handling call
0810 23:33:04.186550 (+ 1890us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:33:04.186644 (+ 94us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:33:05.004652 (+818008us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:05.362382 (+357730us) tablet_replica.cc:205] Creating consensus instance
0810 23:33:05.416392 (+ 54010us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260810 23:33:05.349830 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 114885us
W20260810 23:33:05.466373 12698 leader_election.cc:341] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:05.467046 12698 leader_election.cc:341] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:05.469821 15455 raft_consensus.cc:482] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:05.470311 12770 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 341) took 1046 ms. Trace:
I20260810 23:33:05.470556 15455 raft_consensus.cc:504] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:05.470686 12770 rpcz_store.cc:276] 0810 23:33:04.424061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.424242 (+ 181us) service_pool.cc:227] Handling call
0810 23:33:04.425841 (+ 1599us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:33:04.425946 (+ 105us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:33:04.670673 (+244727us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:05.350440 (+679767us) tablet_replica.cc:205] Creating consensus instance
0810 23:33:05.470238 (+119798us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:33:05.474751 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d491aa5218ca4fb2a496d319a0cac592. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:05.479969 15455 leader_election.cc:288] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:05.491919 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 403caf051714438bb77fc2fbefca15c5) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:05.502233 13276 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:04 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:05.507685 13691 tablet_bootstrap.cc:654] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
W20260810 23:33:05.528260 12700 leader_election.cc:341] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:05.529268 12700 leader_election.cc:302] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:05.533964 13406 heartbeater.cc:503] Master 127.11.226.189:42101 was elected leader, sending a full tablet report...
I20260810 23:33:05.540812 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc0e5cb08c44f7b9662d63905d99b02" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:05.554011 15691 tablet.cc:1722] T 0d1b15dd545c4598aa5c7a9dc53618ec P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260810 23:33:05.557345 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 53115.1us
W20260810 23:33:05.558279 13212 leader_election.cc:341] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:05.559186 13212 leader_election.cc:341] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:05.570297 15683 log.cc:924] Time spent T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260810 23:33:05.571375 15677 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:53280 (request call id 318) took 2000 ms. Trace:
I20260810 23:33:05.571866 15677 rpcz_store.cc:276] 0810 23:33:03.572462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.624992 (+ 52530us) service_pool.cc:227] Handling call
0810 23:33:05.569214 (+1944222us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:03.687750 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 23:33:03.688300 (+ 550us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 23:33:03.688361 (+ 61us) alter_schema_op.cc:65] Acquired schema lock
0810 23:33:03.688457 (+ 96us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 23:33:03.688652 (+ 195us) alter_schema_op.cc:126] START. Timestamp: 1786404783686475
0810 23:33:03.688684 (+ 32us) op_driver.cc:331] REPLICATION: starting
0810 23:33:03.689486 (+ 802us) log.cc:842] Serialized 173 byte log entry
0810 23:33:05.006697 (+1317211us) op_driver.cc:444] REPLICATION: finished
0810 23:33:05.252056 (+245359us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 23:33:05.337449 (+ 85393us) log.cc:842] Serialized 5 byte log entry
0810 23:33:05.507360 (+169911us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:05.510390 (+ 3030us) log.cc:842] Serialized 14 byte log entry
0810 23:33:05.557583 (+ 47193us) alter_schema_op.cc:71] Released schema lock
0810 23:33:05.561672 (+ 4089us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":245091,"prepare.queue_time_us":224,"prepare.run_cpu_time_us":17871,"prepare.run_wall_time_us":91015,"replication_time_us":1317845,"thread_start_us":273864,"threads_started":3,"wal-append.queue_time_us":103465,"wal-append.run_cpu_time_us":1919,"wal-append.run_wall_time_us":1002535}]]}
I20260810 23:33:05.578070 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62705f99687340aaa1aca0942b28487f" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:05.579519 13327 raft_consensus.cc:2379] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:33:05.594947 12235 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_a8675270899846aaa4209e810eb7614b" } new_table_name: "default.Table_5e1f708104aa41118f94a7e35cecf800" modify_external_catalogs: true
W20260810 23:33:05.596088 13212 leader_election.cc:341] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:05.606103 15455 raft_consensus.cc:482] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:05.606920 15455 raft_consensus.cc:504] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:05.563064 14121 ts_tablet_manager.cc:1918] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:05.613946 14121 log.cc:1196] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/10033af79e854103bc47a046d2219ca0
I20260810 23:33:05.622457 14121 ts_tablet_manager.cc:1939] T 10033af79e854103bc47a046d2219ca0 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:33:05.632525 13665 tablet_bootstrap.cc:492] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:05.639321 15586 raft_consensus.cc:482] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:05.661819 13691 tablet_bootstrap.cc:492] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:05.665627 13691 ts_tablet_manager.cc:1397] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.280s user 0.011s sys 0.002s
W20260810 23:33:05.658735 13212 consensus_peers.cc:487] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Couldn't send request to peer f26517150f1c480a877f15aec71d020b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:05.669193 15685 raft_consensus.cc:482] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:05.675535 12755 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:04 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:05.685691 13665 tablet_bootstrap.cc:654] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:05.660430 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41ddf9f6f71443ba75c97d2b010372c" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:05.656780 15586 raft_consensus.cc:504] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:33:05.709288 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 181090us
I20260810 23:33:05.710373 13327 raft_consensus.cc:2454] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:33:05.625365 12802 raft_consensus.cc:3037] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:05.734143 15586 leader_election.cc:288] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:05.737236 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41ddf9f6f71443ba75c97d2b010372c" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:05.739027 13060 raft_consensus.cc:2454] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:05.742514 12952 leader_election.cc:302] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:05.666635 15455 leader_election.cc:288] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:05.750530 13212 leader_election.cc:302] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:05.752419 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 274) took 5688 ms. Trace:
I20260810 23:33:05.752944 14121 rpcz_store.cc:276] 0810 23:33:00.064348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.088774 (+ 24426us) service_pool.cc:227] Handling call
0810 23:33:05.136311 (+5047537us) ts_tablet_manager.cc:1145] Deleting tablet 10033af79e854103bc47a046d2219ca0
0810 23:33:05.136361 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:05.275646 (+139285us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:05.562668 (+287022us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:05.752335 (+189667us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5022103}
I20260810 23:33:05.707837 15486 ts_tablet_manager.cc:1918] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:05.760324 15665 raft_consensus.cc:482] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:05.761969 15665 raft_consensus.cc:504] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:05.764467 15486 log.cc:1196] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/f1e8311c941b43058384cee6d8263dc1
I20260810 23:33:05.772794 15665 leader_election.cc:288] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:05.776193 15455 raft_consensus.cc:2781] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:05.776985 15455 raft_consensus.cc:482] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:05.777833 15455 raft_consensus.cc:3037] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:05.781008 14121 tablet_replica.cc:330] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:33:05.785537 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c39d751e544d9a8afa26dc935a819a" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:33:05.787950 13212 leader_election.cc:341] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:05.469265 15668 raft_consensus.cc:504] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:33:05.765486 12803 consensus_meta.cc:287] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.687s user 0.009s sys 0.006s
W20260810 23:33:05.799136 12958 leader_election.cc:341] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: a41ddf9f6f71443ba75c97d2b010372c
I20260810 23:33:05.795254 15410 raft_consensus.cc:2726] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:05.796551 12803 raft_consensus.cc:2454] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:33:05.815788 15082 raft_consensus.cc:2229] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:05.678824 15685 raft_consensus.cc:504] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:05.766170 15486 ts_tablet_manager.cc:1939] T f1e8311c941b43058384cee6d8263dc1 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:05.787122 15586 raft_consensus.cc:2781] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:05.834056 15676 leader_election.cc:288] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:05.833164 15665 raft_consensus.cc:482] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:05.837188 15665 raft_consensus.cc:504] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:05.828941 15586 raft_consensus.cc:482] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:05.841763 15586 raft_consensus.cc:3037] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:05.843075 15410 consensus_queue.cc:1035] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:33:05.872391 15668 leader_election.cc:288] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:05.873905 15685 leader_election.cc:288] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:05.880923 15082 raft_consensus.cc:2258] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:05.884418 15082 ts_tablet_manager.cc:1905] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:05.885056 15665 leader_election.cc:288] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:05.889483 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240adfa4be844518bbc9624d0e408789" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:33:05.893782 13212 leader_election.cc:341] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:05.909687 13514 tablet_bootstrap.cc:654] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:05.913643 13665 tablet_bootstrap.cc:492] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:05.914487 13665 ts_tablet_manager.cc:1397] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.282s user 0.011s sys 0.015s
I20260810 23:33:05.956144 15586 raft_consensus.cc:504] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:05.961755 13691 raft_consensus.cc:348] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:05.963135 13691 raft_consensus.cc:374] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:05.978746 12802 raft_consensus.cc:2454] T f1452c292e9c476b9e26f56998f0ad55 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:33:05.981302 15455 raft_consensus.cc:504] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:05.987341 15706 tablet.cc:1722] T 0d1b15dd545c4598aa5c7a9dc53618ec P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:33:05.988823 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41ddf9f6f71443ba75c97d2b010372c" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:05.989802 13060 raft_consensus.cc:2379] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in current term 1: Already voted for candidate f26517150f1c480a877f15aec71d020b in this term.
I20260810 23:33:05.951741 13665 raft_consensus.cc:348] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:05.992640 13665 raft_consensus.cc:374] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:05.993031 13665 raft_consensus.cc:729] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:05.993881 13665 consensus_queue.cc:260] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:33:05.995365 15636 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.210s user 0.007s sys 0.005s
I20260810 23:33:05.996547 13665 ts_tablet_manager.cc:1428] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.082s user 0.006s sys 0.005s
I20260810 23:33:06.000329 14121 raft_consensus.cc:2229] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:06.001955 14121 raft_consensus.cc:2258] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:06.005116 14121 ts_tablet_manager.cc:1905] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:06.005743 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41ddf9f6f71443ba75c97d2b010372c" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:33:06.006768 13327 raft_consensus.cc:2379] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:33:06.010335 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 323) took 2340 ms. Trace:
I20260810 23:33:06.010790 15486 rpcz_store.cc:276] 0810 23:33:03.670110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.670530 (+ 420us) service_pool.cc:227] Handling call
0810 23:33:05.049999 (+1379469us) ts_tablet_manager.cc:1145] Deleting tablet f1e8311c941b43058384cee6d8263dc1
0810 23:33:05.156794 (+106795us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:05.405196 (+248402us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:05.707343 (+302147us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:06.010237 (+302894us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1378835}
I20260810 23:33:06.016252 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 321) took 2358 ms. Trace:
I20260810 23:33:06.016921 15486 rpcz_store.cc:276] 0810 23:33:03.658221 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.658728 (+ 507us) service_pool.cc:227] Handling call
0810 23:33:06.015751 (+2357023us) ts_tablet_manager.cc:1145] Deleting tablet 10033af79e854103bc47a046d2219ca0
0810 23:33:06.015883 (+ 132us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.016175 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2343352}
I20260810 23:33:06.019191 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 319) took 2378 ms. Trace:
I20260810 23:33:06.019876 15486 rpcz_store.cc:276] 0810 23:33:03.641006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.641712 (+ 706us) service_pool.cc:227] Handling call
0810 23:33:06.018746 (+2377034us) ts_tablet_manager.cc:1145] Deleting tablet f935119e4d9b4accb79db98185c80aca
0810 23:33:06.018807 (+ 61us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.019127 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1926,"tablet-delete.queue_time_us":2348347}
I20260810 23:33:06.022189 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 324) took 2350 ms. Trace:
I20260810 23:33:06.022706 15486 rpcz_store.cc:276] 0810 23:33:03.671762 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.672424 (+ 662us) service_pool.cc:227] Handling call
0810 23:33:06.021855 (+2349431us) ts_tablet_manager.cc:1145] Deleting tablet 4f3049bcdc6e43c795961c74f0d008f5
0810 23:33:06.021904 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.022127 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2346007}
I20260810 23:33:06.025101 15486 tablet_replica.cc:330] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:06.026613 15455 leader_election.cc:288] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:05.963837 13691 raft_consensus.cc:729] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:06.032089 13691 consensus_queue.cc:260] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:06.034524 13691 ts_tablet_manager.cc:1428] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.299s user 0.004s sys 0.005s
I20260810 23:33:06.027390 15586 leader_election.cc:288] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:06.029462 13665 tablet_bootstrap.cc:492] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:06.062536 13691 tablet_bootstrap.cc:492] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:06.063266 13665 tablet_bootstrap.cc:654] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:06.069886 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 10033af79e854103bc47a046d2219ca0 (table default.Table_620c572fc75b44fe8ca892789875c1f7 [id=cf58351bb8594bdbb92168f093ec91a0]) successfully deleted
I20260810 23:33:06.070811 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet f1e8311c941b43058384cee6d8263dc1 (table default.Table_d8373b61098245bcaab43448a20578f6 [id=7934cc21dac24c74848a668abe1dd606]) successfully deleted
I20260810 23:33:06.091666 13665 tablet_bootstrap.cc:492] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:06.092568 13665 ts_tablet_manager.cc:1397] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.063s user 0.012s sys 0.007s
I20260810 23:33:06.096400 13665 raft_consensus.cc:348] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:33:06.096726 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 10033af79e854103bc47a046d2219ca0 because the tablet was not found. No further retry: Not found: Tablet not found: 10033af79e854103bc47a046d2219ca0
I20260810 23:33:06.097817 13665 raft_consensus.cc:374] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:06.098321 13665 raft_consensus.cc:729] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
W20260810 23:33:06.098359 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet f935119e4d9b4accb79db98185c80aca because the tablet was not found. No further retry: Not found: Tablet not found: f935119e4d9b4accb79db98185c80aca
I20260810 23:33:06.099570 13665 consensus_queue.cc:260] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:06.103163 13665 ts_tablet_manager.cc:1428] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
W20260810 23:33:06.099815 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 4f3049bcdc6e43c795961c74f0d008f5 because the tablet was not found. No further retry: Not found: Tablet not found: 4f3049bcdc6e43c795961c74f0d008f5
I20260810 23:33:06.122133 15486 raft_consensus.cc:2229] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:06.123026 15486 raft_consensus.cc:2258] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:06.126647 15486 ts_tablet_manager.cc:1905] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:06.133955 13665 tablet_bootstrap.cc:492] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:06.145360 12170 master-stress-test.cc:578] Killing test thread 1/10
I20260810 23:33:06.145457 15721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e964330cf113403a8aa5b73ddd3b31d0)
I20260810 23:33:06.146697 15721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
W20260810 23:33:06.157745 12214 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 58010us
I20260810 23:33:06.160108 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 428 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:33:06.167611 13691 tablet_bootstrap.cc:654] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:06.179342 13665 tablet_bootstrap.cc:654] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:06.202670 15721 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:33:06.226526 13665 tablet_bootstrap.cc:492] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:06.227093 13665 ts_tablet_manager.cc:1397] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.093s user 0.010s sys 0.007s
I20260810 23:33:06.230136 13665 raft_consensus.cc:348] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.231034 13665 raft_consensus.cc:374] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:06.231321 13665 raft_consensus.cc:729] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:06.252879 13691 tablet_bootstrap.cc:492] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:06.253960 13691 ts_tablet_manager.cc:1397] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.192s user 0.010s sys 0.005s
I20260810 23:33:06.258912 13665 consensus_queue.cc:260] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.267144 12224 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 0d1b15dd545c4598aa5c7a9dc53618ec (table default.Table_0bf7fdea09a840b9a1394935cc53d643 [id=15edc7d801a94b40b8ab324e25d530e4]). Expected version 1 got 0
W20260810 23:33:06.276362 12700 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.000s sys 0.003s
I20260810 23:33:06.276371 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc0e5cb08c44f7b9662d63905d99b02" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:33:06.276597 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c39d751e544d9a8afa26dc935a819a" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:33:06.276928 12700 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.565s user 0.000s sys 0.003s
I20260810 23:33:06.277334 13336 raft_consensus.cc:2379] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 191d120f4c7f44fc8475305a6d2e75f3 in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:33:06.277269 13327 raft_consensus.cc:2454] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
W20260810 23:33:06.277232 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 565937us
I20260810 23:33:06.278322 12700 leader_election.cc:302] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:06.280180 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3ac117f3b34c1bb55b6a258d4fd539" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:33:06.285962 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet e17e5ec604044047b499312e27ca3de4 (table default.Table_dc485eb27eb64d70ba14404adc87fb9a [id=b27a53844fa3419d80a7313ba2437c1e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e17e5ec604044047b499312e27ca3de4 consensus state has no leader
I20260810 23:33:06.289203 12223 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 391) took 1530 ms. Trace:
I20260810 23:33:06.289602 12223 rpcz_store.cc:276] 0810 23:33:04.758982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.778750 (+ 19768us) service_pool.cc:227] Handling call
0810 23:33:04.783439 (+ 4689us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:33:04.785478 (+ 2039us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:33:06.287776 (+1502298us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:33:06.288347 (+ 571us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:33:06.288938 (+ 591us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:05.497922 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:05.498365 (+ 443us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:05.498390 (+ 25us) write_op.cc:435] Acquired schema lock
0810 23:33:05.498402 (+ 12us) tablet.cc:660] Decoding operations
0810 23:33:05.498712 (+ 310us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:05.498760 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:05.498780 (+ 20us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:05.499002 (+ 222us) tablet.cc:699] Row locks acquired
0810 23:33:05.499015 (+ 13us) write_op.cc:260] PREPARE: finished
0810 23:33:05.499154 (+ 139us) write_op.cc:270] Start()
0810 23:33:05.499248 (+ 94us) write_op.cc:276] Timestamp: P: 1786404785499065 usec, L: 0
0810 23:33:05.499264 (+ 16us) op_driver.cc:331] REPLICATION: starting
0810 23:33:05.500219 (+ 955us) log.cc:842] Serialized 433 byte log entry
0810 23:33:06.179178 (+678959us) op_driver.cc:444] REPLICATION: finished
0810 23:33:06.221543 (+ 42365us) write_op.cc:301] APPLY: starting
0810 23:33:06.221649 (+ 106us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:06.221799 (+ 150us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:06.221946 (+ 147us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:06.222586 (+ 640us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:06.222838 (+ 252us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:33:06.222893 (+ 55us) write_op.cc:312] APPLY: finished
0810 23:33:06.223200 (+ 307us) log.cc:842] Serialized 29 byte log entry
0810 23:33:06.269411 (+ 46211us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.269695 (+ 284us) write_op.cc:454] Released schema lock
0810 23:33:06.269742 (+ 47us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":42021,"num_ops":2,"prepare.queue_time_us":710958,"prepare.run_cpu_time_us":2374,"prepare.run_wall_time_us":2569,"replication_time_us":679752,"thread_start_us":41518,"threads_started":1}]]}
W20260810 23:33:06.295660 12700 leader_election.cc:341] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:06.296427 12700 leader_election.cc:302] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:06.305810 12240 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 392) took 1496 ms. Trace:
I20260810 23:33:06.306304 12240 rpcz_store.cc:276] 0810 23:33:04.808649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:05.297157 (+488508us) service_pool.cc:227] Handling call
0810 23:33:05.340097 (+ 42940us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:33:05.355255 (+ 15158us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:33:06.303776 (+948521us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:33:06.304752 (+ 976us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:33:06.305374 (+ 622us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:05.592813 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:05.593237 (+ 424us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:05.593267 (+ 30us) write_op.cc:435] Acquired schema lock
0810 23:33:05.593280 (+ 13us) tablet.cc:660] Decoding operations
0810 23:33:05.616522 (+ 23242us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:05.616580 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:05.616606 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:05.623397 (+ 6791us) tablet.cc:699] Row locks acquired
0810 23:33:05.623420 (+ 23us) write_op.cc:260] PREPARE: finished
0810 23:33:05.623611 (+ 191us) write_op.cc:270] Start()
0810 23:33:05.623731 (+ 120us) write_op.cc:276] Timestamp: P: 1786404785623558 usec, L: 0
0810 23:33:05.623750 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 23:33:05.640196 (+ 16446us) log.cc:842] Serialized 433 byte log entry
0810 23:33:06.256173 (+615977us) op_driver.cc:444] REPLICATION: finished
0810 23:33:06.300880 (+ 44707us) write_op.cc:301] APPLY: starting
0810 23:33:06.300980 (+ 100us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:06.301171 (+ 191us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:06.301305 (+ 134us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:06.301890 (+ 585us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:06.302064 (+ 174us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:33:06.302120 (+ 56us) write_op.cc:312] APPLY: finished
0810 23:33:06.302380 (+ 260us) log.cc:842] Serialized 29 byte log entry
0810 23:33:06.302575 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.302721 (+ 146us) write_op.cc:454] Released schema lock
0810 23:33:06.302762 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:06.303166 (+ 404us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.303192 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":44480,"apply.run_cpu_time_us":2541,"apply.run_wall_time_us":2541,"mutex_wait_us":65093,"num_ops":2,"prepare.queue_time_us":123063,"prepare.run_cpu_time_us":3336,"prepare.run_wall_time_us":69946,"replication_time_us":632163,"threads_started":1}]]}
W20260810 23:33:06.273882 12958 leader_election.cc:341] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 62705f99687340aaa1aca0942b28487f
I20260810 23:33:06.312999 12958 leader_election.cc:302] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:06.315117 15455 raft_consensus.cc:3037] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:06.330084 13665 ts_tablet_manager.cc:1428] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.103s user 0.010s sys 0.000s
I20260810 23:33:06.331293 13665 tablet_bootstrap.cc:492] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:06.319550 12698 leader_election.cc:302] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
W20260810 23:33:06.334825 12698 leader_election.cc:341] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:06.345491 12236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet dce0ec83ed8943a0b0788505bb75d1df
I20260810 23:33:06.343478 12240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_4b640b4e6af94983b828197e5ece8d86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:06.349960 15455 raft_consensus.cc:2726] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:06.350967 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet dce0ec83ed8943a0b0788505bb75d1df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:05 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:33:06.356711 12235 catalog_manager.cc:5755] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54c9310d98ad4d20b9a584bde8525227 consensus state has no leader
I20260810 23:33:06.357638 12235 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 394) took 1338 ms. Trace:
W20260810 23:33:06.358036 12756 consensus_meta.cc:287] T 00b1e631390b4b69a6ba8c7b1ba5cfef P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 1.294s user 0.017s sys 0.007s
I20260810 23:33:06.360625 12768 tablet_service.cc:1514] Processing DeleteTablet for tablet dce0ec83ed8943a0b0788505bb75d1df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:05 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:06.358057 12235 rpcz_store.cc:276] 0810 23:33:05.019002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:05.503371 (+484369us) service_pool.cc:227] Handling call
0810 23:33:05.594569 (+ 91198us) spinlock_profiling.cc:243] Waited 90 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e77d6 00007f0ce6e2d634 00007f0ce6e2d5c2 00007f0ce6e2d4f1 00007f0ce6e2d46c 00007f0ce6e2c722 00007f0ce50c694c 00007f0ce50c5d86
0810 23:33:05.596343 (+ 1774us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:33:05.597758 (+ 1415us) catalog_manager.cc:3616] Apply alter schema
0810 23:33:05.598593 (+ 835us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:33:05.598854 (+ 261us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:33:06.355494 (+756640us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:33:06.357502 (+ 2008us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:06.167629 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:06.168437 (+ 808us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:06.168480 (+ 43us) write_op.cc:435] Acquired schema lock
0810 23:33:06.168505 (+ 25us) tablet.cc:660] Decoding operations
0810 23:33:06.168946 (+ 441us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:06.169040 (+ 94us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:06.169079 (+ 39us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:06.169311 (+ 232us) tablet.cc:699] Row locks acquired
0810 23:33:06.169335 (+ 24us) write_op.cc:260] PREPARE: finished
0810 23:33:06.169537 (+ 202us) write_op.cc:270] Start()
0810 23:33:06.169711 (+ 174us) write_op.cc:276] Timestamp: P: 1786404786169446 usec, L: 0
0810 23:33:06.169738 (+ 27us) op_driver.cc:331] REPLICATION: starting
0810 23:33:06.170921 (+ 1183us) log.cc:842] Serialized 405 byte log entry
0810 23:33:06.340341 (+169420us) op_driver.cc:444] REPLICATION: finished
0810 23:33:06.350067 (+ 9726us) write_op.cc:301] APPLY: starting
0810 23:33:06.350210 (+ 143us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:06.350422 (+ 212us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:06.350707 (+ 285us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:06.351239 (+ 532us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:06.351594 (+ 355us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:06.351698 (+ 104us) write_op.cc:312] APPLY: finished
0810 23:33:06.354588 (+ 2890us) log.cc:842] Serialized 23 byte log entry
0810 23:33:06.354855 (+ 267us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.354945 (+ 90us) write_op.cc:454] Released schema lock
0810 23:33:06.354988 (+ 43us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":207005952,"child_traces":[["op",{"apply.queue_time_us":9294,"num_ops":1,"prepare.queue_time_us":493231,"prepare.run_cpu_time_us":3683,"prepare.run_wall_time_us":3747,"replication_time_us":170481}]]}
I20260810 23:33:06.365427 12235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "default.Table_b07cd9e36cc84081a4ac019dc2dcad7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:06.331789 12224 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:33:06.370198 12793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c39d751e544d9a8afa26dc935a819a" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:06.371454 12793 raft_consensus.cc:2454] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:06.372282 12236 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 393) took 1563 ms. Trace:
W20260810 23:33:06.372478 12235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b07cd9e36cc84081a4ac019dc2dcad7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:06.377576 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet dce0ec83ed8943a0b0788505bb75d1df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:05 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:06.372673 12236 rpcz_store.cc:276] 0810 23:33:04.809363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:05.423029 (+613666us) service_pool.cc:227] Handling call
0810 23:33:05.428907 (+ 5878us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:33:05.430266 (+ 1359us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:33:05.430488 (+ 222us) catalog_manager.cc:2761] Locking tablets
0810 23:33:05.431216 (+ 728us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:33:06.344106 (+912890us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:33:06.344884 (+ 778us) catalog_manager.cc:2809] Committing in-memory state
0810 23:33:06.345176 (+ 292us) catalog_manager.cc:2813] Aborting table tasks
0810 23:33:06.372150 (+ 26974us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:05.662946 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:05.732160 (+ 69214us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:05.732191 (+ 31us) write_op.cc:435] Acquired schema lock
0810 23:33:05.732206 (+ 15us) tablet.cc:660] Decoding operations
0810 23:33:05.732704 (+ 498us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:05.732749 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:05.732785 (+ 36us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:05.733618 (+ 833us) tablet.cc:699] Row locks acquired
0810 23:33:05.733641 (+ 23us) write_op.cc:260] PREPARE: finished
0810 23:33:05.733814 (+ 173us) write_op.cc:270] Start()
0810 23:33:05.733926 (+ 112us) write_op.cc:276] Timestamp: P: 1786404785733760 usec, L: 0
0810 23:33:05.733947 (+ 21us) op_driver.cc:331] REPLICATION: starting
0810 23:33:06.159070 (+425123us) log.cc:842] Serialized 774 byte log entry
0810 23:33:06.339843 (+180773us) op_driver.cc:444] REPLICATION: finished
0810 23:33:06.341244 (+ 1401us) write_op.cc:301] APPLY: starting
0810 23:33:06.341337 (+ 93us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:06.341596 (+ 259us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:06.341906 (+ 310us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:06.342518 (+ 612us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:06.342729 (+ 211us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:33:06.342799 (+ 70us) write_op.cc:312] APPLY: finished
0810 23:33:06.343114 (+ 315us) log.cc:842] Serialized 29 byte log entry
0810 23:33:06.343329 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.343472 (+ 143us) write_op.cc:454] Released schema lock
0810 23:33:06.343517 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:06.368157 (+ 24640us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.368189 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1111,"apply.run_cpu_time_us":2947,"apply.run_wall_time_us":27262,"mutex_wait_us":81697,"num_ops":2,"prepare.queue_time_us":209476,"prepare.run_cpu_time_us":4966,"prepare.run_wall_time_us":496744,"replication_time_us":605792}]]}
W20260810 23:33:06.381484 12240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b640b4e6af94983b828197e5ece8d86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:06.385932 13691 raft_consensus.cc:348] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:06.387667 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc0e5cb08c44f7b9662d63905d99b02" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:06.388739 12802 raft_consensus.cc:2379] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in current term 1: Already voted for candidate 191d120f4c7f44fc8475305a6d2e75f3 in this term.
I20260810 23:33:06.395772 15296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 428 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:33:06.399675 13665 tablet_bootstrap.cc:654] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:06.405122 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 681674cb24274f10b200f264f155c2d1; no voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:33:06.417447 15689 raft_consensus.cc:2781] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:33:06.417971 15689 raft_consensus.cc:482] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f54aa78a60784a3bb2f5eb4bb2401ce3)
I20260810 23:33:06.418383 15689 raft_consensus.cc:3037] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:33:06.422847 13514 tablet_bootstrap.cc:492] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:33:06.423570 13514 ts_tablet_manager.cc:1397] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 1.223s user 0.012s sys 0.007s
I20260810 23:33:06.429670 15410 raft_consensus.cc:2726] T 422b25457d1549e186641fcedd8feab9 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:06.431677 15410 raft_consensus.cc:2781] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260810 23:33:06.441732 12958 leader_election.cc:341] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: a41ddf9f6f71443ba75c97d2b010372c
I20260810 23:33:06.442459 12958 leader_election.cc:302] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:06.444002 15700 raft_consensus.cc:2726] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:33:06.444674 13691 raft_consensus.cc:374] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:06.445007 13691 raft_consensus.cc:729] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:06.445999 13691 consensus_queue.cc:260] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:06.447782 15410 raft_consensus.cc:482] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:06.448344 15410 raft_consensus.cc:3037] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:06.450014 12756 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 340) took 2031 ms. Trace:
W20260810 23:33:06.454155 15664 lock_manager.cc:586] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 8606919409353307012) tablet 00000000000000000000000000000000 cur holder 0x7b5800021600
I20260810 23:33:06.407181 15721 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260810 23:33:06.487368 13665 tablet_bootstrap.cc:492] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:06.487946 13665 ts_tablet_manager.cc:1397] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.157s user 0.026s sys 0.009s
I20260810 23:33:06.475572 13691 ts_tablet_manager.cc:1428] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.163s user 0.005s sys 0.001s
I20260810 23:33:06.494258 14121 ts_tablet_manager.cc:1918] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:06.498330 14121 log.cc:1196] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/4f3049bcdc6e43c795961c74f0d008f5
I20260810 23:33:06.497283 15486 ts_tablet_manager.cc:1918] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:06.450766 12756 rpcz_store.cc:276] 0810 23:33:04.418026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.475195 (+ 57169us) service_pool.cc:227] Handling call
0810 23:33:04.513842 (+ 38647us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:33:04.513935 (+ 93us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:33:05.063045 (+549110us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:06.359054 (+1296009us) tablet_replica.cc:205] Creating consensus instance
0810 23:33:06.449790 (+ 90736us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:33:06.502344 13514 raft_consensus.cc:348] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.503301 13514 raft_consensus.cc:374] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:06.503630 13514 raft_consensus.cc:729] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
W20260810 23:33:06.504311 15706 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.515s user 0.004s sys 0.000s
I20260810 23:33:06.497908 13691 tablet_bootstrap.cc:492] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:06.499287 15486 log.cc:1196] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/240adfa4be844518bbc9624d0e408789
I20260810 23:33:06.513461 13665 raft_consensus.cc:348] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.514273 13665 raft_consensus.cc:374] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:06.514526 13665 raft_consensus.cc:729] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:06.515208 13665 consensus_queue.cc:260] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.517510 13665 ts_tablet_manager.cc:1428] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.029s user 0.002s sys 0.004s
I20260810 23:33:06.518569 13665 tablet_bootstrap.cc:492] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:06.521677 15586 raft_consensus.cc:482] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:06.522234 15586 raft_consensus.cc:504] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.527271 15586 leader_election.cc:288] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:06.528721 12235 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 396) took 1008 ms. Trace:
I20260810 23:33:06.528759 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3ac117f3b34c1bb55b6a258d4fd539" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:33:06.531399 13212 leader_election.cc:341] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:06.532485 12745 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:47536 (request call id 84) took 1989 ms. Trace:
I20260810 23:33:06.532856 13665 tablet_bootstrap.cc:654] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:06.532974 12745 rpcz_store.cc:276] 0810 23:33:04.521916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.522065 (+ 149us) service_pool.cc:227] Handling call
0810 23:33:05.707363 (+1185298us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54001310d4. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3b7268 00007f3cdf3a36cd 00007f3cddc4be04 00007f3cddc4bd92 00007f3cddc4bcc1 00007f3cddc4bc3c 00007f3cddc4aef2 00007f3cdc16c94c 00007f3cdc16bd86 00007f3cdc16b72f
0810 23:33:05.806491 (+ 99128us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b54000f4e54. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3b7268 00007f3cdf3a36cd 00007f3cddc4be04 00007f3cddc4bd92 00007f3cddc4bcc1 00007f3cddc4bc3c 00007f3cddc4aef2 00007f3cdc16c94c 00007f3cdc16bd86 00007f3cdc16b72f
0810 23:33:06.322679 (+516188us) spinlock_profiling.cc:243] Waited 513 ms on lock 0x7b5400150254. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3b7268 00007f3cdf3a36cd 00007f3cddc4be04 00007f3cddc4bd92 00007f3cddc4bcc1 00007f3cddc4bc3c 00007f3cddc4aef2 00007f3cdc16c94c 00007f3cdc16bd86 00007f3cdc16b72f
0810 23:33:06.511329 (+188650us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1686206464}
I20260810 23:33:06.543954 14121 ts_tablet_manager.cc:1939] T 4f3049bcdc6e43c795961c74f0d008f5 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:33:06.544689 15486 ts_tablet_manager.cc:1939] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:06.546219 12240 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34742 (request call id 395) took 1026 ms. Trace:
I20260810 23:33:06.546728 13141 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:06.529083 12235 rpcz_store.cc:276] 0810 23:33:05.520293 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.364743 (+844450us) service_pool.cc:227] Handling call
0810 23:33:06.373054 (+ 8311us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:33:06.414872 (+ 41818us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:33:06.504737 (+ 89865us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:33:06.528213 (+ 23476us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:33:06.528613 (+ 400us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:06.438510 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:06.438995 (+ 485us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:06.439027 (+ 32us) write_op.cc:435] Acquired schema lock
0810 23:33:06.439042 (+ 15us) tablet.cc:660] Decoding operations
0810 23:33:06.439389 (+ 347us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:06.439473 (+ 84us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:06.439501 (+ 28us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:06.439768 (+ 267us) tablet.cc:699] Row locks acquired
0810 23:33:06.439785 (+ 17us) write_op.cc:260] PREPARE: finished
0810 23:33:06.439932 (+ 147us) write_op.cc:270] Start()
0810 23:33:06.440054 (+ 122us) write_op.cc:276] Timestamp: P: 1786404786439836 usec, L: 0
0810 23:33:06.440075 (+ 21us) op_driver.cc:331] REPLICATION: starting
0810 23:33:06.488816 (+ 48741us) log.cc:842] Serialized 433 byte log entry
0810 23:33:06.489606 (+ 790us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729c471 00007f0ce7313198 00007f0ce7774ade 00007f0ce77742b8 00007f0ce7778ba1 00007f0ce7778b59 00007f0ce7778ae9 00007f0ce7778ab1 00007f0ce7777dad 0000000000580784 00000000005805b9
0810 23:33:06.501918 (+ 12312us) op_driver.cc:444] REPLICATION: finished
0810 23:33:06.502359 (+ 441us) write_op.cc:301] APPLY: starting
0810 23:33:06.502427 (+ 68us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:06.502571 (+ 144us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:06.502595 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:06.503163 (+ 568us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:06.503341 (+ 178us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:33:06.503398 (+ 57us) write_op.cc:312] APPLY: finished
0810 23:33:06.503661 (+ 263us) log.cc:842] Serialized 29 byte log entry
0810 23:33:06.503855 (+ 194us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.503955 (+ 100us) write_op.cc:454] Released schema lock
0810 23:33:06.504004 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:06.504295 (+ 291us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.504320 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":234,"apply.run_cpu_time_us":2159,"apply.run_wall_time_us":2158,"num_ops":2,"prepare.queue_time_us":21794,"prepare.run_cpu_time_us":4101,"prepare.run_wall_time_us":51373,"raft.queue_time_us":107,"raft.run_cpu_time_us":849,"raft.run_wall_time_us":849,"replication_time_us":61750,"spinlock_wait_cycles":110548608}]]}
I20260810 23:33:06.551090 12170 master-stress-test.cc:580] Killed test thread 1/10
I20260810 23:33:06.551389 12170 master-stress-test.cc:578] Killing test thread 2/10
I20260810 23:33:06.552466 13665 tablet_bootstrap.cc:492] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:06.552572 12235 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_a0956ecd557b42e292fde54f05842c47" } new_table_name: "default.Table_cf212e8521974a72a6b86f2ea6ef2ffc" modify_external_catalogs: true
I20260810 23:33:06.553118 13665 ts_tablet_manager.cc:1397] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.005s
I20260810 23:33:06.554751 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 325) took 2882 ms. Trace:
I20260810 23:33:06.555125 15486 rpcz_store.cc:276] 0810 23:33:03.672577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.673474 (+ 897us) service_pool.cc:227] Handling call
0810 23:33:06.024849 (+2351375us) ts_tablet_manager.cc:1145] Deleting tablet 240adfa4be844518bbc9624d0e408789
0810 23:33:06.024931 (+ 82us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.236046 (+211115us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:06.496938 (+260892us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:06.554673 (+ 57735us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70751,"tablet-delete.queue_time_us":2346571}
I20260810 23:33:06.556619 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 282) took 6486 ms. Trace:
I20260810 23:33:06.556975 14121 rpcz_store.cc:276] 0810 23:33:00.069721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.070328 (+ 607us) service_pool.cc:227] Handling call
0810 23:33:05.780769 (+5710441us) ts_tablet_manager.cc:1145] Deleting tablet 4f3049bcdc6e43c795961c74f0d008f5
0810 23:33:05.780847 (+ 78us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.231178 (+450331us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:06.493940 (+262762us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:06.556544 (+ 62604us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5663455}
I20260810 23:33:06.558573 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 4f3049bcdc6e43c795961c74f0d008f5 (table default.Table_7548a45d866747b9b0944e13b211c643 [id=c6af3828fca045bdbe16438171ff83cd]) successfully deleted
I20260810 23:33:06.546614 12240 rpcz_store.cc:276] 0810 23:33:05.519639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.313831 (+794192us) service_pool.cc:227] Handling call
0810 23:33:06.343216 (+ 29385us) spinlock_profiling.cc:243] Waited 26.2 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e7156 00007f0ce6e24034 00007f0ce6e23fc2 00007f0ce6e23ef1 00007f0ce6e23e6c 00007f0ce6e23122 00007f0ce50c694c 00007f0ce50c5d86
0810 23:33:06.468105 (+124889us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:33:06.471852 (+ 3747us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:33:06.544207 (+ 72355us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:33:06.545833 (+ 1626us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:33:06.546143 (+ 310us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:06.490088 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:06.490563 (+ 475us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:06.490587 (+ 24us) write_op.cc:435] Acquired schema lock
0810 23:33:06.490601 (+ 14us) tablet.cc:660] Decoding operations
0810 23:33:06.490895 (+ 294us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:06.490934 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:06.490956 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:06.491243 (+ 287us) tablet.cc:699] Row locks acquired
0810 23:33:06.491258 (+ 15us) write_op.cc:260] PREPARE: finished
0810 23:33:06.491388 (+ 130us) write_op.cc:270] Start()
0810 23:33:06.491488 (+ 100us) write_op.cc:276] Timestamp: P: 1786404786491290 usec, L: 0
0810 23:33:06.491505 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:33:06.492167 (+ 662us) log.cc:842] Serialized 433 byte log entry
0810 23:33:06.533792 (+ 41625us) op_driver.cc:444] REPLICATION: finished
0810 23:33:06.534213 (+ 421us) write_op.cc:301] APPLY: starting
0810 23:33:06.534282 (+ 69us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:06.534395 (+ 113us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:06.534510 (+ 115us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:06.535055 (+ 545us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:06.535238 (+ 183us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:33:06.535288 (+ 50us) write_op.cc:312] APPLY: finished
0810 23:33:06.535535 (+ 247us) log.cc:842] Serialized 29 byte log entry
0810 23:33:06.535735 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.535824 (+ 89us) write_op.cc:454] Released schema lock
0810 23:33:06.535868 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:06.536110 (+ 242us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.536133 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":60234112,"child_traces":[["op",{"apply.queue_time_us":238,"apply.run_cpu_time_us":2099,"apply.run_wall_time_us":2098,"num_ops":2,"prepare.queue_time_us":15913,"prepare.run_cpu_time_us":2286,"prepare.run_wall_time_us":2339,"replication_time_us":42200}]]}
I20260810 23:33:06.559027 13665 raft_consensus.cc:348] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.560173 13665 raft_consensus.cc:374] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:06.560494 13665 raft_consensus.cc:729] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:06.561873 12170 master-stress-test.cc:580] Killed test thread 2/10
I20260810 23:33:06.562043 13691 tablet_bootstrap.cc:654] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:06.562168 12170 master-stress-test.cc:578] Killing test thread 3/10
I20260810 23:33:06.562449 12170 master-stress-test.cc:580] Killed test thread 3/10
I20260810 23:33:06.562649 12170 master-stress-test.cc:578] Killing test thread 4/10
I20260810 23:33:06.562959 12170 master-stress-test.cc:580] Killed test thread 4/10
I20260810 23:33:06.563174 12170 master-stress-test.cc:578] Killing test thread 5/10
I20260810 23:33:06.566087 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 240adfa4be844518bbc9624d0e408789 (table default.Table_32d7e9316d054deca361c6e3d9b5e1f4 [id=43ef82e91d4a4e2b8bb06d4cab61c8c4]) successfully deleted
I20260810 23:33:06.565798 13665 consensus_queue.cc:260] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.506209 13514 consensus_queue.cc:260] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.568744 13665 ts_tablet_manager.cc:1428] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260810 23:33:06.572588 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 283) took 6234 ms. Trace:
I20260810 23:33:06.578313 14121 rpcz_store.cc:276] 0810 23:33:00.337815 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.342710 (+ 4895us) service_pool.cc:227] Handling call
0810 23:33:06.568273 (+6225563us) ts_tablet_manager.cc:1145] Deleting tablet 422b25457d1549e186641fcedd8feab9
0810 23:33:06.572102 (+ 3829us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.572502 (+ 400us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6218062}
I20260810 23:33:06.572714 12240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_9bbb5f34eb6d4d65995978e1c125a8bd" } modify_external_catalogs: true
I20260810 23:33:06.581131 12240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "default.Table_9bbb5f34eb6d4d65995978e1c125a8bd" } modify_external_catalogs: true
I20260810 23:33:06.581415 15486 tablet_replica.cc:330] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:06.582151 13665 tablet_bootstrap.cc:492] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:06.582906 15486 raft_consensus.cc:2229] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:06.586588 13213 leader_election.cc:302] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:33:06.587966 13213 leader_election.cc:302] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:33:06.592585 15586 raft_consensus.cc:2781] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:06.583691 15486 raft_consensus.cc:2258] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:06.587642 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3ac117f3b34c1bb55b6a258d4fd539" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:06.588310 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c39d751e544d9a8afa26dc935a819a" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:06.591684 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc0e5cb08c44f7b9662d63905d99b02" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:06.602553 13665 tablet_bootstrap.cc:654] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:06.608830 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 286) took 5146 ms. Trace:
I20260810 23:33:06.609556 13514 ts_tablet_manager.cc:1428] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.186s user 0.011s sys 0.000s
I20260810 23:33:06.611321 14121 rpcz_store.cc:276] 0810 23:33:01.462368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:01.465418 (+ 3050us) service_pool.cc:227] Handling call
0810 23:33:06.602681 (+5137263us) ts_tablet_manager.cc:1145] Deleting tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd
0810 23:33:06.602742 (+ 61us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.608748 (+ 6006us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5136420}
I20260810 23:33:06.617097 13514 tablet_bootstrap.cc:492] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
W20260810 23:33:06.619086 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd already in progress: creating tablet
I20260810 23:33:06.625003 15586 raft_consensus.cc:686] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:33:06.626281 15586 consensus_queue.cc:237] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:06.619171 15744 raft_consensus.cc:2781] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
W20260810 23:33:06.628325 13213 leader_election.cc:341] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: a41ddf9f6f71443ba75c97d2b010372c
I20260810 23:33:06.628923 13213 leader_election.cc:302] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:06.630041 15486 ts_tablet_manager.cc:1905] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:06.631108 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 288) took 5168 ms. Trace:
I20260810 23:33:06.634586 14121 rpcz_store.cc:276] 0810 23:33:01.463082 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:01.463192 (+ 110us) service_pool.cc:227] Handling call
0810 23:33:06.630677 (+5167485us) ts_tablet_manager.cc:1145] Deleting tablet 32fad37a28e14903b8bde5aac6ee4e69
0810 23:33:06.630727 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.631037 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5162946}
I20260810 23:33:06.636096 15745 consensus_queue.cc:1035] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:33:06.637785 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 32fad37a28e14903b8bde5aac6ee4e69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32fad37a28e14903b8bde5aac6ee4e69 already in progress: creating tablet
I20260810 23:33:06.635610 13691 tablet_bootstrap.cc:492] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:06.640475 13691 ts_tablet_manager.cc:1397] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.143s user 0.012s sys 0.000s
I20260810 23:33:06.642818 15410 raft_consensus.cc:504] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.636436 15700 raft_consensus.cc:482] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:06.644330 15700 raft_consensus.cc:504] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:06.646492 13691 raft_consensus.cc:348] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:06.647199 13691 raft_consensus.cc:374] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:06.647341 15700 leader_election.cc:288] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:33:06.651512 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b9d20d09c4d1ba9a4b10959807e50" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:06.652460 12806 raft_consensus.cc:2379] T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:33:06.652433 13691 raft_consensus.cc:729] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:06.657550 13691 consensus_queue.cc:260] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:06.602648 15689 raft_consensus.cc:504] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.641189 14121 tablet_replica.cc:330] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:33:06.662358 13691 ts_tablet_manager.cc:1428] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.021s user 0.001s sys 0.005s
I20260810 23:33:06.630162 15748 raft_consensus.cc:2726] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:33:06.665555 14121 raft_consensus.cc:2229] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:06.666363 14121 raft_consensus.cc:2258] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:06.668274 15689 leader_election.cc:288] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 2 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:06.668160 13691 tablet_bootstrap.cc:492] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:06.681180 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b9d20d09c4d1ba9a4b10959807e50" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:06.688661 12952 leader_election.cc:302] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:06.690337 15700 raft_consensus.cc:3037] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:06.628557 15744 raft_consensus.cc:686] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:33:06.693430 14121 ts_tablet_manager.cc:1905] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:06.711838 13691 tablet_bootstrap.cc:654] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:06.715659 13514 tablet_bootstrap.cc:654] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:06.721966 15700 raft_consensus.cc:2726] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:06.734135 13691 tablet_bootstrap.cc:492] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:06.734788 13691 ts_tablet_manager.cc:1397] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.005s
I20260810 23:33:06.737740 13691 raft_consensus.cc:348] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:06.738559 13691 raft_consensus.cc:374] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:06.738853 13691 raft_consensus.cc:729] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:06.739660 13691 consensus_queue.cc:260] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:06.741895 13691 ts_tablet_manager.cc:1428] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260810 23:33:06.754588 13691 tablet_bootstrap.cc:492] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:06.756207 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 4dffbb8214c64ed0b456d00519d6bfb9 (DEFAULT_TABLE table=default.Table_13d7888530b048f38e069ee640677b1c [id=bc3dacd88f7e4fa78fbefdb197df991b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:06.757398 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dffbb8214c64ed0b456d00519d6bfb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:06.771920 13691 tablet_bootstrap.cc:654] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:06.722075 15410 leader_election.cc:288] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:06.752158 15689 raft_consensus.cc:482] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:06.775671 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c53785fdd24df79f138df6176e75d2" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:06.776739 15748 raft_consensus.cc:482] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:06.777292 15748 raft_consensus.cc:504] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:06.777454 12228 catalog_manager.cc:5495] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } }
I20260810 23:33:06.778025 15689 raft_consensus.cc:504] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:06.779045 12228 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]). Expected version 1 got 0
I20260810 23:33:06.779592 15748 leader_election.cc:288] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:06.781952 15689 leader_election.cc:288] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:33:06.784140 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1f24e226b94ccf8ad5a656c80a5b3b" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:33:06.784570 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 154360us
I20260810 23:33:06.784834 13059 raft_consensus.cc:2454] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:06.786502 12240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 7293fddc208a4ee7ad85906f28f3be08
W20260810 23:33:06.788272 13213 leader_election.cc:341] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 240adfa4be844518bbc9624d0e408789
I20260810 23:33:06.788659 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 7293fddc208a4ee7ad85906f28f3be08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:06 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:06.790225 13213 leader_election.cc:302] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:06.790639 15082 ts_tablet_manager.cc:1918] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:06.790896 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 7293fddc208a4ee7ad85906f28f3be08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:06 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:06.791076 15082 log.cc:1196] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/a480093c481941f0bb35949c7606f37f
I20260810 23:33:06.692235 15744 consensus_queue.cc:237] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.791719 12240 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34742 (request call id 398) took 1184 ms. Trace:
I20260810 23:33:06.792064 15082 ts_tablet_manager.cc:1939] T a480093c481941f0bb35949c7606f37f P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
W20260810 23:33:06.792073 13213 raft_consensus.cc:2670] T 240adfa4be844518bbc9624d0e408789 P f54aa78a60784a3bb2f5eb4bb2401ce3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260810 23:33:06.793076 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 294) took 5802 ms. Trace:
I20260810 23:33:06.792057 12240 rpcz_store.cc:276] 0810 23:33:05.607401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.572218 (+964817us) service_pool.cc:227] Handling call
0810 23:33:06.581700 (+ 9482us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:33:06.582349 (+ 649us) catalog_manager.cc:2752] Modifying in-memory table state
0810 23:33:06.582566 (+ 217us) catalog_manager.cc:2761] Locking tablets
0810 23:33:06.740911 (+158345us) catalog_manager.cc:2775] Removing table and tablets from system table
0810 23:33:06.785988 (+ 45077us) catalog_manager.cc:2793] Removing table from by-name map
0810 23:33:06.786270 (+ 282us) catalog_manager.cc:2809] Committing in-memory state
0810 23:33:06.786393 (+ 123us) catalog_manager.cc:2813] Aborting table tasks
0810 23:33:06.791629 (+ 5236us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:06.744878 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:06.745324 (+ 446us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:06.745353 (+ 29us) write_op.cc:435] Acquired schema lock
0810 23:33:06.745369 (+ 16us) tablet.cc:660] Decoding operations
0810 23:33:06.745757 (+ 388us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:06.745805 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:06.745831 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:06.746145 (+ 314us) tablet.cc:699] Row locks acquired
0810 23:33:06.746162 (+ 17us) write_op.cc:260] PREPARE: finished
0810 23:33:06.746295 (+ 133us) write_op.cc:270] Start()
0810 23:33:06.746386 (+ 91us) write_op.cc:276] Timestamp: P: 1786404786746237 usec, L: 0
0810 23:33:06.746405 (+ 19us) op_driver.cc:331] REPLICATION: starting
0810 23:33:06.746997 (+ 592us) log.cc:842] Serialized 734 byte log entry
0810 23:33:06.782462 (+ 35465us) op_driver.cc:444] REPLICATION: finished
0810 23:33:06.783057 (+ 595us) write_op.cc:301] APPLY: starting
0810 23:33:06.783133 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:06.783253 (+ 120us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:06.783353 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:06.783671 (+ 318us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:06.783825 (+ 154us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0810 23:33:06.783867 (+ 42us) write_op.cc:312] APPLY: finished
0810 23:33:06.784219 (+ 352us) log.cc:842] Serialized 29 byte log entry
0810 23:33:06.784386 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.784470 (+ 84us) write_op.cc:454] Released schema lock
0810 23:33:06.784505 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:06.784700 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.784727 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":412,"apply.run_cpu_time_us":1833,"apply.run_wall_time_us":1879,"num_ops":2,"prepare.queue_time_us":1752,"prepare.run_cpu_time_us":2375,"prepare.run_wall_time_us":2376,"replication_time_us":35967}]]}
I20260810 23:33:06.796914 14121 ts_tablet_manager.cc:1918] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260810 23:33:06.797420 14121 log.cc:1196] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/88c53785fdd24df79f138df6176e75d2
I20260810 23:33:06.798293 14121 ts_tablet_manager.cc:1939] T 88c53785fdd24df79f138df6176e75d2 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:33:06.800475 13213 leader_election.cc:302] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
W20260810 23:33:06.800434 12700 leader_election.cc:341] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:06.802093 12700 leader_election.cc:302] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:06.805140 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 77b4478f4bc34dabb6e7cbff282442a4 (DEFAULT_TABLE table=default.Table_4884ea66b39f44e99d957b62d1f38e0c [id=bc5681871c97482881fcbb19b32cf6f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:06.806499 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b4478f4bc34dabb6e7cbff282442a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:06.809005 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c39d751e544d9a8afa26dc935a819a" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:06.812867 13212 leader_election.cc:302] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:06.813771 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet 7293fddc208a4ee7ad85906f28f3be08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:06 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:06.815068 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 4dffbb8214c64ed0b456d00519d6bfb9 (DEFAULT_TABLE table=default.Table_13d7888530b048f38e069ee640677b1c [id=bc3dacd88f7e4fa78fbefdb197df991b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:06.816501 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dffbb8214c64ed0b456d00519d6bfb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:06.800988 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 4dffbb8214c64ed0b456d00519d6bfb9 (DEFAULT_TABLE table=default.Table_13d7888530b048f38e069ee640677b1c [id=bc3dacd88f7e4fa78fbefdb197df991b]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:33:06.803190 13213 leader_election.cc:341] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: a41ddf9f6f71443ba75c97d2b010372c
I20260810 23:33:06.793460 15082 rpcz_store.cc:276] 0810 23:33:00.990617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:00.990741 (+ 124us) service_pool.cc:227] Handling call
0810 23:33:05.092476 (+4101735us) ts_tablet_manager.cc:1145] Deleting tablet a480093c481941f0bb35949c7606f37f
0810 23:33:05.092535 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:05.833149 (+740614us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b5400150754. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3c5352 00007f3cde7db26c 00007f3cdf3e8cfa 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49 00007f3cdf3ffd11 00007f3cdf3fef3d 0000000000580784
0810 23:33:06.534357 (+701208us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:06.690519 (+156162us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:06.793002 (+102483us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47336,"spinlock_wait_cycles":1466957056,"tablet-delete.queue_time_us":4101123}
I20260810 23:33:06.832569 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dffbb8214c64ed0b456d00519d6bfb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:06.832595 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 294) took 4841 ms. Trace:
I20260810 23:33:06.833003 14121 rpcz_store.cc:276] 0810 23:33:01.991490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:01.993718 (+ 2228us) service_pool.cc:227] Handling call
0810 23:33:06.641012 (+4647294us) ts_tablet_manager.cc:1145] Deleting tablet 88c53785fdd24df79f138df6176e75d2
0810 23:33:06.641063 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.724538 (+ 83475us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:06.796425 (+ 71887us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:06.832516 (+ 36091us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4632911}
I20260810 23:33:06.834563 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 295) took 4829 ms. Trace:
I20260810 23:33:06.834532 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 88c53785fdd24df79f138df6176e75d2 (table default.Table_d0b8422dca5e40ff919741fcee7d547c [id=71c9424f327040a68889fc257935c535]) successfully deleted
I20260810 23:33:06.834872 14121 rpcz_store.cc:276] 0810 23:33:02.005717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:02.006210 (+ 493us) service_pool.cc:227] Handling call
0810 23:33:06.834085 (+4827875us) ts_tablet_manager.cc:1145] Deleting tablet 8163921fdc2c442cb335041c79de6d6d
0810 23:33:06.834138 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.834507 (+ 369us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4817603}
W20260810 23:33:06.836719 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 8163921fdc2c442cb335041c79de6d6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8163921fdc2c442cb335041c79de6d6d already in progress: creating tablet
I20260810 23:33:06.837257 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 298) took 4619 ms. Trace:
I20260810 23:33:06.837558 14121 rpcz_store.cc:276] 0810 23:33:02.217796 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:02.217932 (+ 136us) service_pool.cc:227] Handling call
0810 23:33:06.835846 (+4617914us) ts_tablet_manager.cc:1145] Deleting tablet 40eb874e0b334c46961e16d8d5aaaaa7
0810 23:33:06.835888 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.837190 (+ 1302us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4591855}
I20260810 23:33:06.839473 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 299) took 4583 ms. Trace:
W20260810 23:33:06.839495 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 40eb874e0b334c46961e16d8d5aaaaa7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40eb874e0b334c46961e16d8d5aaaaa7 already in progress: creating tablet
I20260810 23:33:06.839813 14121 rpcz_store.cc:276] 0810 23:33:02.255826 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:02.266534 (+ 10708us) service_pool.cc:227] Handling call
0810 23:33:06.838927 (+4572393us) ts_tablet_manager.cc:1145] Deleting tablet 7031c257fdf34a28b6954d3d4827a8da
0810 23:33:06.838974 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.839410 (+ 436us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4571457}
I20260810 23:33:06.841125 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 300) took 4485 ms. Trace:
I20260810 23:33:06.841442 14121 rpcz_store.cc:276] 0810 23:33:02.355864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:02.363265 (+ 7401us) service_pool.cc:227] Handling call
0810 23:33:06.840739 (+4477474us) ts_tablet_manager.cc:1145] Deleting tablet dfda619f19a5459494b1e4f6d22821cc
0810 23:33:06.840781 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.841071 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4476805}
W20260810 23:33:06.842182 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 7031c257fdf34a28b6954d3d4827a8da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7031c257fdf34a28b6954d3d4827a8da already in progress: creating tablet
I20260810 23:33:06.842679 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 311) took 3858 ms. Trace:
I20260810 23:33:06.842988 14121 rpcz_store.cc:276] 0810 23:33:02.984698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:02.984807 (+ 109us) service_pool.cc:227] Handling call
0810 23:33:06.842291 (+3857484us) ts_tablet_manager.cc:1145] Deleting tablet f47d8e462d814a15a82fd0d5218e2d0e
0810 23:33:06.842333 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.842599 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3856788}
W20260810 23:33:06.843480 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet dfda619f19a5459494b1e4f6d22821cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfda619f19a5459494b1e4f6d22821cc already in progress: creating tablet
I20260810 23:33:06.843986 14121 tablet_replica.cc:330] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
W20260810 23:33:06.844769 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet f47d8e462d814a15a82fd0d5218e2d0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f47d8e462d814a15a82fd0d5218e2d0e already in progress: creating tablet
I20260810 23:33:06.844882 14121 raft_consensus.cc:2229] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:06.845742 14121 raft_consensus.cc:2258] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:06.848712 15689 raft_consensus.cc:2726] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:33:06.851424 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 51022.4us
W20260810 23:33:06.852310 12700 leader_election.cc:341] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:06.853205 12700 leader_election.cc:341] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:06.853574 12700 leader_election.cc:302] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:06.854290 15410 raft_consensus.cc:2726] T afc0e5cb08c44f7b9662d63905d99b02 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260810 23:33:06.856405 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 43630.3us
I20260810 23:33:06.859162 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 77b4478f4bc34dabb6e7cbff282442a4 (DEFAULT_TABLE table=default.Table_4884ea66b39f44e99d957b62d1f38e0c [id=bc5681871c97482881fcbb19b32cf6f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:06.858958 14121 ts_tablet_manager.cc:1905] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:06.862341 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 75656f6e1d274da1be7bfa9d43fa45c8 (DEFAULT_TABLE table=default.Table_523cb26bd2ea40f1b341060ff15317d3 [id=ef51c5adb1694c948f0430dfd66c967d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:06.863689 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75656f6e1d274da1be7bfa9d43fa45c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:06.864270 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet a480093c481941f0bb35949c7606f37f (table default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f]) successfully deleted
I20260810 23:33:06.869578 13514 tablet_bootstrap.cc:492] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:33:06.870100 13514 ts_tablet_manager.cc:1397] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.253s user 0.008s sys 0.008s
I20260810 23:33:06.873594 13514 raft_consensus.cc:348] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.874675 13514 raft_consensus.cc:374] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:06.875118 13514 raft_consensus.cc:729] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:33:06.875847 13514 consensus_queue.cc:260] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.877104 15586 raft_consensus.cc:2781] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:06.877758 15586 raft_consensus.cc:482] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:06.878118 15586 raft_consensus.cc:3037] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:06.865597 15757 raft_consensus.cc:482] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:06.880553 15757 raft_consensus.cc:504] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.883050 15757 leader_election.cc:288] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:06.884289 12240 master_service.cc:946] ReplaceTablet: received request to replace tablet 46450c84a9054fbdbb5523ec835d43d2 from {username='slave'} at 127.0.0.1:34742
I20260810 23:33:06.858991 15745 raft_consensus.cc:2781] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:06.884850 15745 raft_consensus.cc:482] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:06.885244 15745 raft_consensus.cc:3037] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:06.890275 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975a9bab969e46d2b939fc76ce350b99" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:06.890303 13280 tablet_service.cc:1467] Processing CreateTablet for tablet 77b4478f4bc34dabb6e7cbff282442a4 (DEFAULT_TABLE table=default.Table_4884ea66b39f44e99d957b62d1f38e0c [id=bc5681871c97482881fcbb19b32cf6f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:06.779793 12228 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]). Expected version 1 got 0
W20260810 23:33:06.891978 13212 leader_election.cc:341] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:06.900527 12755 tablet_service.cc:1467] Processing CreateTablet for tablet 75656f6e1d274da1be7bfa9d43fa45c8 (DEFAULT_TABLE table=default.Table_523cb26bd2ea40f1b341060ff15317d3 [id=ef51c5adb1694c948f0430dfd66c967d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:06.900590 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 75656f6e1d274da1be7bfa9d43fa45c8 (DEFAULT_TABLE table=default.Table_523cb26bd2ea40f1b341060ff15317d3 [id=ef51c5adb1694c948f0430dfd66c967d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:06.906558 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b4478f4bc34dabb6e7cbff282442a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:06.899760 12228 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (table default.Table_4a7751c5b61c4332adeeb9ee00474b27 [id=2c8320255aab4036b5350d082f9e04ef]). Expected version 2 got 0
I20260810 23:33:06.910533 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75656f6e1d274da1be7bfa9d43fa45c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:06.910591 12228 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:33:06.920233 13514 ts_tablet_manager.cc:1428] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20260810 23:33:06.921957 13025 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0212a1965a4ba5ae1673b5b4df3392 (DEFAULT_TABLE table=default.Table_64b9e1ba04bf4003bc78a1f24cdbd715 [id=179a899b06c54cd888f52bf9ee665890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:06.923522 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0212a1965a4ba5ae1673b5b4df3392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:06.925894 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0212a1965a4ba5ae1673b5b4df3392 (DEFAULT_TABLE table=default.Table_64b9e1ba04bf4003bc78a1f24cdbd715 [id=179a899b06c54cd888f52bf9ee665890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:06.926373 13023 tablet_service.cc:1467] Processing CreateTablet for tablet 620100d53e564096b127a46ab690e3de (DEFAULT_TABLE table=default.Table_35fbbca76d6c43419846a96203e57279 [id=01a719299d5b4f518bfd8d59a7ab211f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:06.927671 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620100d53e564096b127a46ab690e3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:06.933373 14121 ts_tablet_manager.cc:1918] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:33:06.933833 14121 log.cc:1196] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/ab8d987109ba40198645881b449cb972
I20260810 23:33:06.934616 14121 ts_tablet_manager.cc:1939] T ab8d987109ba40198645881b449cb972 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:33:06.938993 13665 tablet_bootstrap.cc:492] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:06.939670 13665 ts_tablet_manager.cc:1397] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.358s user 0.009s sys 0.009s
I20260810 23:33:06.949314 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 312) took 3478 ms. Trace:
I20260810 23:33:06.949760 14121 rpcz_store.cc:276] 0810 23:33:03.470753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.470904 (+ 151us) service_pool.cc:227] Handling call
0810 23:33:06.843810 (+3372906us) ts_tablet_manager.cc:1145] Deleting tablet ab8d987109ba40198645881b449cb972
0810 23:33:06.843849 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.880074 (+ 36225us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:06.933085 (+ 53011us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:06.949217 (+ 16132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3372092}
I20260810 23:33:06.951467 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 316) took 3479 ms. Trace:
I20260810 23:33:06.951866 14121 rpcz_store.cc:276] 0810 23:33:03.471995 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.472762 (+ 767us) service_pool.cc:227] Handling call
0810 23:33:06.951067 (+3478305us) ts_tablet_manager.cc:1145] Deleting tablet ca0dd19112504f8ca57f64dcf0bd50a8
0810 23:33:06.951129 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.951396 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3477246}
I20260810 23:33:06.953132 14121 tablet_replica.cc:330] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:33:06.954417 14121 raft_consensus.cc:2229] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:06.954593 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0212a1965a4ba5ae1673b5b4df3392 (DEFAULT_TABLE table=default.Table_64b9e1ba04bf4003bc78a1f24cdbd715 [id=179a899b06c54cd888f52bf9ee665890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:06.954944 14121 raft_consensus.cc:2258] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:06.955833 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0212a1965a4ba5ae1673b5b4df3392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:06.958782 13665 raft_consensus.cc:348] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.959722 13665 raft_consensus.cc:374] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:06.959990 13665 raft_consensus.cc:729] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:06.960938 13665 consensus_queue.cc:260] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:06.963300 13665 ts_tablet_manager.cc:1428] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260810 23:33:06.953398 15748 raft_consensus.cc:482] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:06.965032 15748 raft_consensus.cc:504] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:06.967013 15748 leader_election.cc:288] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:06.968087 13665 tablet_bootstrap.cc:492] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:06.975138 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet ab8d987109ba40198645881b449cb972 (table default.Table_d8a0e3e655b9417d9df3ff6d82b76f15 [id=d0b86eafe9544cd98d8fce8ece3b5c10]) successfully deleted
W20260810 23:33:06.976202 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet ca0dd19112504f8ca57f64dcf0bd50a8 because the tablet was not found. No further retry: Not found: Tablet not found: ca0dd19112504f8ca57f64dcf0bd50a8
I20260810 23:33:06.979810 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 297) took 5924 ms. Trace:
I20260810 23:33:06.980840 13691 tablet_bootstrap.cc:492] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:06.981335 13691 ts_tablet_manager.cc:1397] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.227s user 0.012s sys 0.000s
I20260810 23:33:06.983023 14121 ts_tablet_manager.cc:1905] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:06.983985 12228 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 0d1b15dd545c4598aa5c7a9dc53618ec (table default.Table_0bf7fdea09a840b9a1394935cc53d643 [id=15edc7d801a94b40b8ab324e25d530e4]). Expected version 1 got 0
I20260810 23:33:06.986464 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8c0d1dacc344cbacb0c5988f0e2221" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:33:06.989274 13212 leader_election.cc:341] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:06.993044 12240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 46450c84a9054fbdbb5523ec835d43d2
I20260810 23:33:06.998077 12763 tablet_service.cc:1514] Processing DeleteTablet for tablet 46450c84a9054fbdbb5523ec835d43d2 with delete_type TABLET_DATA_DELETED (replaced by tablet c9c84a370e7740649dab114582ebb4ab) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:06.980571 15082 rpcz_store.cc:276] 0810 23:33:01.055134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:01.055289 (+ 155us) service_pool.cc:227] Handling call
0810 23:33:06.867628 (+5812339us) ts_tablet_manager.cc:1145] Deleting tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd
0810 23:33:06.979047 (+111419us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:06.979674 (+ 627us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5811698}
I20260810 23:33:06.998660 13012 tablet_service.cc:1514] Processing DeleteTablet for tablet 46450c84a9054fbdbb5523ec835d43d2 with delete_type TABLET_DATA_DELETED (replaced by tablet c9c84a370e7740649dab114582ebb4ab) from {username='slave'} at 127.0.0.1:58598
W20260810 23:33:06.999785 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd already in progress: creating tablet
I20260810 23:33:07.001355 13514 tablet_bootstrap.cc:492] T bd921e48f22348639f7549d47b165185 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:33:07.005195 15586 raft_consensus.cc:504] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
W20260810 23:33:07.005086 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 147248us
I20260810 23:33:07.006897 13691 raft_consensus.cc:348] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:07.007683 13691 raft_consensus.cc:374] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 23:33:07.007220 12700 leader_election.cc:341] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: Tablet not RUNNING: STOPPED
W20260810 23:33:07.006624 15760 log.cc:924] Time spent T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
I20260810 23:33:07.007939 13691 raft_consensus.cc:729] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:07.031700 13691 consensus_queue.cc:260] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:07.034723 13665 tablet_bootstrap.cc:654] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
W20260810 23:33:07.039196 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 181238us
I20260810 23:33:07.039945 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b4478f4bc34dabb6e7cbff282442a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:07.040366 13213 leader_election.cc:302] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:07.042832 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0212a1965a4ba5ae1673b5b4df3392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:07.043864 13691 ts_tablet_manager.cc:1428] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.062s user 0.006s sys 0.000s
I20260810 23:33:07.044050 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75656f6e1d274da1be7bfa9d43fa45c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:07.044888 13691 tablet_bootstrap.cc:492] T a480093c481941f0bb35949c7606f37f P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:07.047808 15757 raft_consensus.cc:2726] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:33:07.049893 15586 leader_election.cc:288] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:07.050894 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1f24e226b94ccf8ad5a656c80a5b3b" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:07.051748 13059 raft_consensus.cc:3037] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:07.054432 12240 catalog_manager.cc:6565] ReplaceTablet: tablet 46450c84a9054fbdbb5523ec835d43d2 deleted and replaced by tablet c9c84a370e7740649dab114582ebb4ab
I20260810 23:33:07.059695 13059 raft_consensus.cc:2454] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:33:07.051304 15745 raft_consensus.cc:504] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.063839 15745 leader_election.cc:288] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:07.080353 13277 tablet_service.cc:1514] Processing DeleteTablet for tablet 46450c84a9054fbdbb5523ec835d43d2 with delete_type TABLET_DATA_DELETED (replaced by tablet c9c84a370e7740649dab114582ebb4ab) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:07.081240 15764 raft_consensus.cc:482] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:07.081708 15764 raft_consensus.cc:504] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.083869 15764 leader_election.cc:288] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:07.090641 13406 heartbeater.cc:503] Master 127.11.226.189:42101 was elected leader, sending a full tablet report...
I20260810 23:33:07.091478 15486 ts_tablet_manager.cc:1918] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:07.095851 15486 log.cc:1196] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/62705f99687340aaa1aca0942b28487f
I20260810 23:33:07.100111 12223 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:33:07.096786 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c39d751e544d9a8afa26dc935a819a" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:33:07.104377 13335 raft_consensus.cc:2379] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 191d120f4c7f44fc8475305a6d2e75f3 in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:33:07.097852 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8c0d1dacc344cbacb0c5988f0e2221" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:07.103204 13277 tablet_service.cc:1467] Processing CreateTablet for tablet 853848cdcdb6431a9d714785ef049aa5 (DEFAULT_TABLE table=default.Table_02075f5963604d95b8d962f1071ebe64 [id=b8a33141d6c648649115fa6daff6d13e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:07.105330 13336 raft_consensus.cc:2454] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:33:07.105894 13277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 853848cdcdb6431a9d714785ef049aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:07.106822 15486 ts_tablet_manager.cc:1939] T 62705f99687340aaa1aca0942b28487f P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:07.107728 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 326) took 3431 ms. Trace:
I20260810 23:33:07.064527 15763 raft_consensus.cc:482] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:07.112438 15763 raft_consensus.cc:504] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:07.114616 15763 leader_election.cc:288] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:07.081296 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c39d751e544d9a8afa26dc935a819a" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:07.082964 13057 raft_consensus.cc:3037] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:07.108786 15745 raft_consensus.cc:482] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:07.118628 15745 raft_consensus.cc:504] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.094189 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806fe5162aef4d36ad5e2ac324047426" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:07.121548 15745 leader_election.cc:288] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:07.122560 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7bd9b6ca4845d8b8705ff970c8f06e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:07.124804 15486 rpcz_store.cc:276] 0810 23:33:03.676203 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.676446 (+ 243us) service_pool.cc:227] Handling call
0810 23:33:06.567820 (+2891374us) ts_tablet_manager.cc:1145] Deleting tablet 62705f99687340aaa1aca0942b28487f
0810 23:33:06.581083 (+ 13263us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:07.010052 (+428969us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:07.091176 (+ 81124us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:07.107665 (+ 16489us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2884840}
I20260810 23:33:07.128149 15486 tablet_replica.cc:330] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:07.128823 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 62705f99687340aaa1aca0942b28487f (table default.Table_96019e28037b4d38aa9fc933a29c8884 [id=dd7ca129aac0430ba386c53a30052148]) successfully deleted
I20260810 23:33:07.129562 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f68708a0b74a58b69581e8324a33f6" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:07.130972 15486 raft_consensus.cc:2229] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:07.131281 13691 tablet_bootstrap.cc:654] T a480093c481941f0bb35949c7606f37f P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:07.131574 15486 raft_consensus.cc:2258] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:07.103322 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 620100d53e564096b127a46ab690e3de (DEFAULT_TABLE table=default.Table_35fbbca76d6c43419846a96203e57279 [id=01a719299d5b4f518bfd8d59a7ab211f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:07.133189 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620100d53e564096b127a46ab690e3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:07.134029 15486 ts_tablet_manager.cc:1905] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:33:07.128903 13212 leader_election.cc:341] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:07.136538 13212 leader_election.cc:341] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:07.137118 13212 leader_election.cc:341] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:07.137698 13025 tablet_service.cc:1467] Processing CreateTablet for tablet 853848cdcdb6431a9d714785ef049aa5 (DEFAULT_TABLE table=default.Table_02075f5963604d95b8d962f1071ebe64 [id=b8a33141d6c648649115fa6daff6d13e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:07.139518 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 853848cdcdb6431a9d714785ef049aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:07.147831 12236 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]). Expected version 1 got 0
I20260810 23:33:07.149449 12236 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (table default.Table_4a7751c5b61c4332adeeb9ee00474b27 [id=2c8320255aab4036b5350d082f9e04ef]). Expected version 2 got 0
I20260810 23:33:07.137656 13212 leader_election.cc:302] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:07.160100 15745 raft_consensus.cc:2781] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:07.160621 15745 raft_consensus.cc:686] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:33:07.161521 15745 consensus_queue.cc:237] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:07.165788 14121 ts_tablet_manager.cc:1918] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:07.166291 14121 log.cc:1196] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/f1e8311c941b43058384cee6d8263dc1
I20260810 23:33:07.167218 14121 ts_tablet_manager.cc:1939] T f1e8311c941b43058384cee6d8263dc1 P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:33:07.168438 15586 raft_consensus.cc:482] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:07.168910 15586 raft_consensus.cc:504] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.171895 15586 leader_election.cc:288] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:07.174206 13691 tablet_bootstrap.cc:492] T a480093c481941f0bb35949c7606f37f P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:07.176326 13691 ts_tablet_manager.cc:1397] T a480093c481941f0bb35949c7606f37f P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.132s user 0.006s sys 0.006s
W20260810 23:33:07.177520 12698 leader_election.cc:341] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 88c53785fdd24df79f138df6176e75d2
I20260810 23:33:07.177992 12698 leader_election.cc:302] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:07.177857 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fad37a28e14903b8bde5aac6ee4e69" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:07.179793 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 8008dbbf09904a2c8d4b9b952cbb2480 (DEFAULT_TABLE table=default.Table_98320d53af774eabb5d826d4c0732dad [id=652ca05bcdf547cdb178a2f5c91eaf03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:07.181272 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8008dbbf09904a2c8d4b9b952cbb2480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:07.184305 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 8008dbbf09904a2c8d4b9b952cbb2480 (DEFAULT_TABLE table=default.Table_98320d53af774eabb5d826d4c0732dad [id=652ca05bcdf547cdb178a2f5c91eaf03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:07.186089 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8008dbbf09904a2c8d4b9b952cbb2480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:07.187357 12236 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:33:07.189801 12236 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]). Expected version 1 got 0
I20260810 23:33:07.197382 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 300) took 6025 ms. Trace:
I20260810 23:33:07.202337 15082 rpcz_store.cc:276] 0810 23:33:01.172510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:01.172672 (+ 162us) service_pool.cc:227] Handling call
0810 23:33:07.003072 (+5830400us) ts_tablet_manager.cc:1145] Deleting tablet 32fad37a28e14903b8bde5aac6ee4e69
0810 23:33:07.109332 (+106260us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:07.197312 (+ 87980us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5829501}
I20260810 23:33:07.201668 12763 tablet_service.cc:1467] Processing CreateTablet for tablet 620100d53e564096b127a46ab690e3de (DEFAULT_TABLE table=default.Table_35fbbca76d6c43419846a96203e57279 [id=01a719299d5b4f518bfd8d59a7ab211f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:07.213817 15082 tablet_replica.cc:330] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:33:07.214951 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 314) took 3743 ms. Trace:
I20260810 23:33:07.182941 13691 raft_consensus.cc:348] T a480093c481941f0bb35949c7606f37f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:07.216495 13691 raft_consensus.cc:374] T a480093c481941f0bb35949c7606f37f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:07.216787 13691 raft_consensus.cc:729] T a480093c481941f0bb35949c7606f37f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:07.217842 13665 tablet_bootstrap.cc:492] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:07.218298 13665 ts_tablet_manager.cc:1397] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.250s user 0.009s sys 0.020s
W20260810 23:33:07.215801 13212 leader_election.cc:341] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:07.227556 12235 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34742 (request call id 397) took 1644 ms. Trace:
I20260810 23:33:07.215391 14121 rpcz_store.cc:276] 0810 23:33:03.471394 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.473482 (+ 2088us) service_pool.cc:227] Handling call
0810 23:33:06.952909 (+3479427us) ts_tablet_manager.cc:1145] Deleting tablet f1e8311c941b43058384cee6d8263dc1
0810 23:33:06.952966 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:07.082013 (+129047us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:07.165437 (+ 83424us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:07.214852 (+ 49415us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3478724}
I20260810 23:33:07.227244 13691 consensus_queue.cc:260] T a480093c481941f0bb35949c7606f37f P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:07.186002 15689 consensus_queue.cc:1035] T 684caf5aaa8347e8b4e02323318fb0b7 P 191d120f4c7f44fc8475305a6d2e75f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20260810 23:33:07.227916 12235 rpcz_store.cc:276] 0810 23:33:05.582792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.552123 (+969331us) service_pool.cc:227] Handling call
0810 23:33:06.553335 (+ 1212us) catalog_manager.cc:2347] Looking up, locking, and authorizing table
0810 23:33:06.554102 (+ 767us) catalog_manager.cc:3616] Apply alter schema
0810 23:33:06.554747 (+ 645us) catalog_manager.cc:3639] Acquired catalog manager lock
0810 23:33:06.555007 (+ 260us) catalog_manager.cc:3786] Updating metadata on disk
0810 23:33:06.698734 (+143727us) catalog_manager.cc:3817] Committing alterations to in-memory state
0810 23:33:07.227460 (+528726us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:06.582908 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:06.583371 (+ 463us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:06.583403 (+ 32us) write_op.cc:435] Acquired schema lock
0810 23:33:06.583419 (+ 16us) tablet.cc:660] Decoding operations
0810 23:33:06.583753 (+ 334us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:06.583809 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:06.583834 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:06.584050 (+ 216us) tablet.cc:699] Row locks acquired
0810 23:33:06.584066 (+ 16us) write_op.cc:260] PREPARE: finished
0810 23:33:06.584215 (+ 149us) write_op.cc:270] Start()
0810 23:33:06.584451 (+ 236us) write_op.cc:276] Timestamp: P: 1786404786584171 usec, L: 0
0810 23:33:06.584473 (+ 22us) op_driver.cc:331] REPLICATION: starting
0810 23:33:06.585175 (+ 702us) log.cc:842] Serialized 405 byte log entry
0810 23:33:06.695881 (+110706us) op_driver.cc:444] REPLICATION: finished
0810 23:33:06.696383 (+ 502us) write_op.cc:301] APPLY: starting
0810 23:33:06.696454 (+ 71us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:06.696567 (+ 113us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:06.696675 (+ 108us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:06.696884 (+ 209us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:06.697044 (+ 160us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:06.697089 (+ 45us) write_op.cc:312] APPLY: finished
0810 23:33:06.697300 (+ 211us) log.cc:842] Serialized 23 byte log entry
0810 23:33:06.697469 (+ 169us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.697533 (+ 64us) write_op.cc:454] Released schema lock
0810 23:33:06.697574 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:06.697779 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.697803 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":301,"apply.run_cpu_time_us":1613,"apply.run_wall_time_us":1612,"num_ops":1,"prepare.queue_time_us":2381,"prepare.run_cpu_time_us":2618,"prepare.run_wall_time_us":2617,"raft.queue_time_us":1554,"raft.run_cpu_time_us":784,"raft.run_wall_time_us":785,"replication_time_us":111281}]]}
I20260810 23:33:07.231698 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 315) took 3760 ms. Trace:
I20260810 23:33:07.232630 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet f1e8311c941b43058384cee6d8263dc1 (table default.Table_d8373b61098245bcaab43448a20578f6 [id=7934cc21dac24c74848a668abe1dd606]) successfully deleted
I20260810 23:33:07.233104 12170 master-stress-test.cc:580] Killed test thread 5/10
I20260810 23:33:07.233433 12170 master-stress-test.cc:578] Killing test thread 6/10
I20260810 23:33:07.233747 12170 master-stress-test.cc:580] Killed test thread 6/10
I20260810 23:33:07.233992 12170 master-stress-test.cc:578] Killing test thread 7/10
I20260810 23:33:07.234992 12170 master-stress-test.cc:580] Killed test thread 7/10
I20260810 23:33:07.235265 12170 master-stress-test.cc:578] Killing test thread 8/10
I20260810 23:33:07.235713 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620100d53e564096b127a46ab690e3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:07.235736 12170 master-stress-test.cc:580] Killed test thread 8/10
I20260810 23:33:07.235975 12170 master-stress-test.cc:578] Killing test thread 9/10
W20260810 23:33:07.237668 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 32fad37a28e14903b8bde5aac6ee4e69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32fad37a28e14903b8bde5aac6ee4e69 already in progress: creating tablet
I20260810 23:33:07.240846 15687 raft_consensus.cc:2726] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 23:33:07.241609 12794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7bd9b6ca4845d8b8705ff970c8f06e" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:07.244294 15082 raft_consensus.cc:2229] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:07.239986 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 8008dbbf09904a2c8d4b9b952cbb2480 (DEFAULT_TABLE table=default.Table_98320d53af774eabb5d826d4c0732dad [id=652ca05bcdf547cdb178a2f5c91eaf03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:07.246354 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8008dbbf09904a2c8d4b9b952cbb2480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:07.249688 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 853848cdcdb6431a9d714785ef049aa5 (DEFAULT_TABLE table=default.Table_02075f5963604d95b8d962f1071ebe64 [id=b8a33141d6c648649115fa6daff6d13e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:07.251070 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 853848cdcdb6431a9d714785ef049aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:07.232157 14121 rpcz_store.cc:276] 0810 23:33:03.471697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.474266 (+ 2569us) service_pool.cc:227] Handling call
0810 23:33:07.231294 (+3757028us) ts_tablet_manager.cc:1145] Deleting tablet 92b81e7056a441d2854ba5ab6b4c5433
0810 23:33:07.231362 (+ 68us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:07.231620 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3756397}
W20260810 23:33:07.253227 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 92b81e7056a441d2854ba5ab6b4c5433 because the tablet was not found. No further retry: Not found: Tablet not found: 92b81e7056a441d2854ba5ab6b4c5433
I20260810 23:33:07.261775 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 324) took 3235 ms. Trace:
I20260810 23:33:07.262189 14121 rpcz_store.cc:276] 0810 23:33:04.026022 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.026196 (+ 174us) service_pool.cc:227] Handling call
0810 23:33:07.260798 (+3234602us) ts_tablet_manager.cc:1145] Deleting tablet 95622c5782e443b9903c85f64852eaa9
0810 23:33:07.260887 (+ 89us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:07.261662 (+ 775us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3225754}
W20260810 23:33:07.263984 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 95622c5782e443b9903c85f64852eaa9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95622c5782e443b9903c85f64852eaa9 already in progress: creating tablet
W20260810 23:33:07.009225 12700 leader_election.cc:341] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:07.269541 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 260425us
I20260810 23:33:07.270581 13665 raft_consensus.cc:348] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.274448 13514 tablet_bootstrap.cc:654] T bd921e48f22348639f7549d47b165185 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:07.280761 13665 raft_consensus.cc:374] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:07.281172 13665 raft_consensus.cc:729] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:07.282670 13665 consensus_queue.cc:260] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.263653 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 329) took 2670 ms. Trace:
I20260810 23:33:07.289343 14121 rpcz_store.cc:276] 0810 23:33:04.592845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.594202 (+ 1357us) service_pool.cc:227] Handling call
0810 23:33:07.263216 (+2669014us) ts_tablet_manager.cc:1145] Deleting tablet d53029158b0142078d66bad842f0a69e
0810 23:33:07.263269 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:07.263591 (+ 322us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2654814}
I20260810 23:33:07.289680 15082 raft_consensus.cc:2258] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:07.291780 14121 tablet_replica.cc:330] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b: stopping tablet replica
W20260810 23:33:07.291786 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet d53029158b0142078d66bad842f0a69e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d53029158b0142078d66bad842f0a69e already in progress: creating tablet
I20260810 23:33:07.301020 13665 ts_tablet_manager.cc:1428] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.082s user 0.010s sys 0.000s
I20260810 23:33:07.307782 12794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3ac117f3b34c1bb55b6a258d4fd539" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:07.308621 12794 raft_consensus.cc:2454] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:07.317133 13665 tablet_bootstrap.cc:492] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:07.329313 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:33:07.347129 13691 ts_tablet_manager.cc:1428] T a480093c481941f0bb35949c7606f37f P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.170s user 0.007s sys 0.000s
I20260810 23:33:07.348503 13691 tablet_bootstrap.cc:492] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:07.355181 15486 ts_tablet_manager.cc:1918] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:07.364822 13691 tablet_bootstrap.cc:654] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:07.366873 15082 ts_tablet_manager.cc:1905] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:07.370968 13665 tablet_bootstrap.cc:654] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:07.375171 15486 log.cc:1196] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/a41ddf9f6f71443ba75c97d2b010372c
I20260810 23:33:07.376507 15455 raft_consensus.cc:482] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:07.377898 15486 ts_tablet_manager.cc:1939] T a41ddf9f6f71443ba75c97d2b010372c P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:07.377388 15455 raft_consensus.cc:504] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } }
I20260810 23:33:07.379097 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 327) took 3702 ms. Trace:
I20260810 23:33:07.379721 15486 rpcz_store.cc:276] 0810 23:33:03.676765 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.676923 (+ 158us) service_pool.cc:227] Handling call
0810 23:33:07.127925 (+3451002us) ts_tablet_manager.cc:1145] Deleting tablet a41ddf9f6f71443ba75c97d2b010372c
0810 23:33:07.127987 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:07.287517 (+159530us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:07.354762 (+ 67245us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:07.378994 (+ 24232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3440413}
W20260810 23:33:07.383177 12952 leader_election.cc:341] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Not found: Tablet not found: 240adfa4be844518bbc9624d0e408789
I20260810 23:33:07.384876 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 564cbf98360d4ceb8bebde7faf0ff673 (DEFAULT_TABLE table=default.Table_3ae548119a244f178ea7e8222ab60395 [id=f315e4339efb4e61b98a5923ca97c823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:07.386092 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564cbf98360d4ceb8bebde7faf0ff673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:07.389535 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975a9bab969e46d2b939fc76ce350b99" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:07.390343 12806 raft_consensus.cc:2454] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:07.390708 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 564cbf98360d4ceb8bebde7faf0ff673 (DEFAULT_TABLE table=default.Table_3ae548119a244f178ea7e8222ab60395 [id=f315e4339efb4e61b98a5923ca97c823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:07.393801 15689 raft_consensus.cc:482] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:07.394796 15689 raft_consensus.cc:504] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.397069 15689 leader_election.cc:288] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:07.385902 15455 leader_election.cc:288] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:07.408313 14121 raft_consensus.cc:2229] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:07.410475 14121 raft_consensus.cc:2258] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:07.421159 13665 tablet_bootstrap.cc:492] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:07.421763 13665 ts_tablet_manager.cc:1397] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.110s user 0.014s sys 0.008s
I20260810 23:33:07.389739 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet a41ddf9f6f71443ba75c97d2b010372c (table default.Table_73f2b1b207fb45f2adecbd37412c5e6f [id=c57f7c273a9841c1ac02523a733a383f]) successfully deleted
I20260810 23:33:07.424578 13665 raft_consensus.cc:348] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.425355 13665 raft_consensus.cc:374] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:07.425637 13665 raft_consensus.cc:729] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:07.426548 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 00f68708a0b74a58b69581e8324a33f6 (table default.Table_d134a55028664442b08b229593877973 [id=26be70827ca442a4b24a26015ffd952a]). Expected version 2 got 0
I20260810 23:33:07.427641 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (table default.Table_4a7751c5b61c4332adeeb9ee00474b27 [id=2c8320255aab4036b5350d082f9e04ef]). Expected version 2 got 0
I20260810 23:33:07.428333 13691 tablet_bootstrap.cc:492] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:07.428834 13691 ts_tablet_manager.cc:1397] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.081s user 0.017s sys 0.003s
I20260810 23:33:07.431535 13665 consensus_queue.cc:260] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.442822 13665 ts_tablet_manager.cc:1428] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260810 23:33:07.443943 13691 raft_consensus.cc:348] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.444790 13691 raft_consensus.cc:374] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:07.445052 13691 raft_consensus.cc:729] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:07.445911 13691 consensus_queue.cc:260] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.454160 13691 ts_tablet_manager.cc:1428] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
W20260810 23:33:07.454833 15664 lock_manager.cc:586] Waited 2 seconds to obtain row lock on key '<redacted>' (key hash 8606919409353307012) tablet 00000000000000000000000000000000 cur holder 0x7b5800021600
I20260810 23:33:07.455369 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c39d751e544d9a8afa26dc935a819a" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:07.456393 12805 raft_consensus.cc:2379] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in current term 1: Already voted for candidate 191d120f4c7f44fc8475305a6d2e75f3 in this term.
I20260810 23:33:07.457019 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8c0d1dacc344cbacb0c5988f0e2221" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:07.457739 12803 raft_consensus.cc:2454] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:07.459002 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 564cbf98360d4ceb8bebde7faf0ff673 (DEFAULT_TABLE table=default.Table_3ae548119a244f178ea7e8222ab60395 [id=f315e4339efb4e61b98a5923ca97c823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:07.460484 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564cbf98360d4ceb8bebde7faf0ff673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:07.464481 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806fe5162aef4d36ad5e2ac324047426" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:07.465350 12806 raft_consensus.cc:2454] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:07.467335 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]). Expected version 1 got 0
I20260810 23:33:07.478895 13691 tablet_bootstrap.cc:492] T 975a9bab969e46d2b939fc76ce350b99 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:07.491014 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 564cbf98360d4ceb8bebde7faf0ff673. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:07.495117 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 315762us
I20260810 23:33:07.495999 13691 tablet_bootstrap.cc:654] T 975a9bab969e46d2b939fc76ce350b99 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:07.496757 12698 leader_election.cc:302] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:07.505381 12698 leader_election.cc:302] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:07.513667 13665 tablet_bootstrap.cc:492] T ba8d56858471421a9fbc361ee6aff28f P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:07.402241 15786 tablet.cc:1722] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:33:07.516593 15486 tablet_replica.cc:330] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:07.518376 15486 raft_consensus.cc:2229] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:07.519780 15486 pending_rounds.cc:62] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3: Trying to abort 1 pending ops.
I20260810 23:33:07.537408 15410 raft_consensus.cc:2726] T 27c39d751e544d9a8afa26dc935a819a P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:33:07.531703 15486 pending_rounds.cc:68] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317114008226279424 op_type: NO_OP noop_request { }
I20260810 23:33:07.540642 15486 raft_consensus.cc:2866] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:33:07.541110 12235 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet c5794637d5dd43f2b05edd53865f4e99 (table default.Table_cf212e8521974a72a6b86f2ea6ef2ffc [id=339a844c37ee460cb948d7e58264e5c8]). Expected version 1 got 0
I20260810 23:33:07.541498 15486 raft_consensus.cc:2258] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:07.542389 12958 leader_election.cc:302] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:07.543972 15455 raft_consensus.cc:2726] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:07.544350 12235 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:33:07.545492 12235 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]). Expected version 1 got 0
I20260810 23:33:07.546092 12235 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (table default.Table_4a7751c5b61c4332adeeb9ee00474b27 [id=2c8320255aab4036b5350d082f9e04ef]). Expected version 2 got 0
W20260810 23:33:07.547365 12214 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 54c9310d98ad4d20b9a584bde8525227 (table default.Table_5e1f708104aa41118f94a7e35cecf800 [id=9e4c719f0c7a4f839aed37bc54648a36], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 54c9310d98ad4d20b9a584bde8525227 consensus state has no leader
I20260810 23:33:07.548756 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f53aca31ee4e338d9b5e281557c121" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:07.553386 15486 ts_tablet_manager.cc:1905] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:07.565766 13665 tablet_bootstrap.cc:654] T ba8d56858471421a9fbc361ee6aff28f P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:07.567214 12236 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_0bf7fdea09a840b9a1394935cc53d643 [id=15edc7d801a94b40b8ab324e25d530e4] alter complete (version 1)
I20260810 23:33:07.572414 12237 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_970b3cb3161f4f5a9fec0f3dfa72f1a5 [id=963512beef0d4063b8e6a5738abd1d92] alter complete (version 3)
I20260810 23:33:07.587555 12797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f68708a0b74a58b69581e8324a33f6" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:07.588711 12797 raft_consensus.cc:2454] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 0.
I20260810 23:33:07.592760 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fad37a28e14903b8bde5aac6ee4e69" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:07.565042 15689 raft_consensus.cc:2781] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:07.595376 15689 raft_consensus.cc:482] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:07.595837 15689 raft_consensus.cc:3037] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:07.600296 15745 raft_consensus.cc:482] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:07.600982 15745 raft_consensus.cc:504] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.605340 15745 leader_election.cc:288] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:07.606211 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd921e48f22348639f7549d47b165185" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:33:07.609161 13212 leader_election.cc:341] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:07.619427 15795 tablet.cc:1722] T c5794637d5dd43f2b05edd53865f4e99 P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260810 23:33:07.623245 12958 leader_election.cc:341] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 240adfa4be844518bbc9624d0e408789
I20260810 23:33:07.629729 12958 leader_election.cc:302] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:07.630697 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet a41ddf9f6f71443ba75c97d2b010372c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:07.631392 15700 raft_consensus.cc:2726] T 240adfa4be844518bbc9624d0e408789 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:07.633898 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 240adfa4be844518bbc9624d0e408789 with delete_type TABLET_DATA_DELETED (replaced by tablet 4344987a6c90428fbdf50765a4631455) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:07.635646 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e8311c941b43058384cee6d8263dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:07.625741 15796 tablet.cc:1722] T c5794637d5dd43f2b05edd53865f4e99 P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260810 23:33:07.643150 12234 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 40) took 4014 ms (client timeout 1000 ms). Trace:
I20260810 23:33:07.650055 12760 tablet_service.cc:1514] Processing DeleteTablet for tablet f935119e4d9b4accb79db98185c80aca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:58 UTC) from {username='slave'} at 127.0.0.1:47546
W20260810 23:33:07.652936 15782 log.cc:924] Time spent T 4d1b9d20d09c4d1ba9a4b10959807e50 P 191d120f4c7f44fc8475305a6d2e75f3: Append to log took a long time: real 0.287s user 0.001s sys 0.000s
I20260810 23:33:07.653183 13691 tablet_bootstrap.cc:492] T 975a9bab969e46d2b939fc76ce350b99 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:07.655233 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet a480093c481941f0bb35949c7606f37f with delete_type TABLET_DATA_DELETED (replaced by tablet 0703cf71490c4c71a95f55863583a03c) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:07.662112 13691 ts_tablet_manager.cc:1397] T 975a9bab969e46d2b939fc76ce350b99 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.183s user 0.017s sys 0.002s
I20260810 23:33:07.670871 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd921e48f22348639f7549d47b165185" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:07.684890 12760 tablet_service.cc:1514] Processing DeleteTablet for tablet a480093c481941f0bb35949c7606f37f with delete_type TABLET_DATA_DELETED (replaced by tablet 0703cf71490c4c71a95f55863583a03c) from {username='slave'} at 127.0.0.1:47546
W20260810 23:33:07.687789 12234 rpcz_store.cc:269] 0810 23:33:03.627752 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.627930 (+ 178us) service_pool.cc:227] Handling call
0810 23:33:07.642318 (+4014388us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":200076,"reported_tablets":24}
I20260810 23:33:07.690752 13059 raft_consensus.cc:3037] T a480093c481941f0bb35949c7606f37f P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:33:07.665793 12228 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 42) took 2337 ms (client timeout 1000 ms). Trace:
W20260810 23:33:07.677084 12226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 69) took 3068 ms (client timeout 1000 ms). Trace:
I20260810 23:33:07.701054 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 62705f99687340aaa1aca0942b28487f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:48 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:33:07.700263 12226 rpcz_store.cc:269] 0810 23:33:04.608666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.608838 (+ 172us) service_pool.cc:227] Handling call
0810 23:33:07.676800 (+3067962us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:05.500603 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:05.501031 (+ 428us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:05.501057 (+ 26us) write_op.cc:435] Acquired schema lock
0810 23:33:05.501070 (+ 13us) tablet.cc:660] Decoding operations
0810 23:33:05.501365 (+ 295us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:05.501403 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:05.501425 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:05.501592 (+ 167us) tablet.cc:699] Row locks acquired
0810 23:33:05.501608 (+ 16us) write_op.cc:260] PREPARE: finished
0810 23:33:05.501744 (+ 136us) write_op.cc:270] Start()
0810 23:33:05.501851 (+ 107us) write_op.cc:276] Timestamp: P: 1786404785501656 usec, L: 0
0810 23:33:05.501871 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:33:05.592348 (+ 90477us) log.cc:842] Serialized 536 byte log entry
0810 23:33:06.221443 (+629095us) op_driver.cc:444] REPLICATION: finished
0810 23:33:06.225718 (+ 4275us) write_op.cc:301] APPLY: starting
0810 23:33:06.226010 (+ 292us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:06.226136 (+ 126us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:06.226289 (+ 153us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:06.226538 (+ 249us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:06.226699 (+ 161us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:06.226748 (+ 49us) write_op.cc:312] APPLY: finished
0810 23:33:06.227013 (+ 265us) log.cc:842] Serialized 23 byte log entry
0810 23:33:06.227214 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.227300 (+ 86us) write_op.cc:454] Released schema lock
0810 23:33:06.227350 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:06.300361 (+ 73011us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:06.300392 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":34373,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":3971,"apply.run_cpu_time_us":2374,"apply.run_wall_time_us":74956,"num_ops":1,"prepare.queue_time_us":100485,"prepare.run_cpu_time_us":2537,"prepare.run_wall_time_us":92065,"replication_time_us":719324,"thread_start_us":10445,"threads_started":1}]]}
I20260810 23:33:07.703523 15764 consensus_queue.cc:1035] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:33:07.704246 13691 raft_consensus.cc:348] T 975a9bab969e46d2b939fc76ce350b99 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.707614 13141 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:07.708101 13514 tablet_bootstrap.cc:492] T bd921e48f22348639f7549d47b165185 P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:33:07.708585 13514 ts_tablet_manager.cc:1397] T bd921e48f22348639f7549d47b165185 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.708s user 0.011s sys 0.006s
I20260810 23:33:07.710395 13691 raft_consensus.cc:374] T 975a9bab969e46d2b939fc76ce350b99 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:07.710677 13691 raft_consensus.cc:729] T 975a9bab969e46d2b939fc76ce350b99 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
W20260810 23:33:07.693415 12228 rpcz_store.cc:269] 0810 23:33:05.328092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:05.365014 (+ 36922us) service_pool.cc:227] Handling call
0810 23:33:07.058771 (+1693757us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:07.058801 (+ 30us) write_op.cc:454] Released schema lock
0810 23:33:07.665701 (+606900us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:07.021555 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:07.021936 (+ 381us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:07.021968 (+ 32us) write_op.cc:435] Acquired schema lock
0810 23:33:07.021980 (+ 12us) tablet.cc:660] Decoding operations
0810 23:33:07.022258 (+ 278us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:07.022301 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:07.022322 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:07.022497 (+ 175us) tablet.cc:699] Row locks acquired
0810 23:33:07.022510 (+ 13us) write_op.cc:260] PREPARE: finished
0810 23:33:07.022635 (+ 125us) write_op.cc:270] Start()
0810 23:33:07.022727 (+ 92us) write_op.cc:276] Timestamp: P: 1786404787022561 usec, L: 0
0810 23:33:07.022741 (+ 14us) op_driver.cc:331] REPLICATION: starting
0810 23:33:07.023277 (+ 536us) log.cc:842] Serialized 536 byte log entry
0810 23:33:07.033755 (+ 10478us) op_driver.cc:444] REPLICATION: finished
0810 23:33:07.034122 (+ 367us) write_op.cc:301] APPLY: starting
0810 23:33:07.034182 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:07.034293 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:07.034412 (+ 119us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:07.034655 (+ 243us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:07.034807 (+ 152us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:07.034854 (+ 47us) write_op.cc:312] APPLY: finished
0810 23:33:07.035066 (+ 212us) log.cc:842] Serialized 23 byte log entry
0810 23:33:07.035240 (+ 174us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:07.035313 (+ 73us) write_op.cc:454] Released schema lock
0810 23:33:07.035348 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":33798,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":207,"apply.run_cpu_time_us":1467,"apply.run_wall_time_us":1466,"num_ops":1,"prepare.queue_time_us":121,"prepare.run_cpu_time_us":2035,"prepare.run_wall_time_us":2037,"raft.queue_time_us":1368,"raft.run_cpu_time_us":708,"raft.run_wall_time_us":708,"replication_time_us":10928}]]}
I20260810 23:33:07.712963 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8c0d1dacc344cbacb0c5988f0e2221" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:33:07.679313 12237 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 68) took 4191 ms (client timeout 1000 ms). Trace:
W20260810 23:33:07.714216 12237 rpcz_store.cc:269] 0810 23:33:03.487565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.504608 (+ 17043us) service_pool.cc:227] Handling call
0810 23:33:03.542070 (+ 37462us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:33:07.679156 (+4137086us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:04.615724 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:04.616188 (+ 464us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:04.616212 (+ 24us) write_op.cc:435] Acquired schema lock
0810 23:33:04.616223 (+ 11us) tablet.cc:660] Decoding operations
0810 23:33:04.616488 (+ 265us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:04.616532 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:04.616561 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:04.616703 (+ 142us) tablet.cc:699] Row locks acquired
0810 23:33:04.616717 (+ 14us) write_op.cc:260] PREPARE: finished
0810 23:33:04.616829 (+ 112us) write_op.cc:270] Start()
0810 23:33:04.616909 (+ 80us) write_op.cc:276] Timestamp: P: 1786404784616790 usec, L: 0
0810 23:33:04.616923 (+ 14us) op_driver.cc:331] REPLICATION: starting
0810 23:33:04.745488 (+128565us) log.cc:842] Serialized 356 byte log entry
0810 23:33:04.921641 (+176153us) op_driver.cc:444] REPLICATION: finished
0810 23:33:04.922124 (+ 483us) write_op.cc:301] APPLY: starting
0810 23:33:04.922265 (+ 141us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:04.922471 (+ 206us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:04.922696 (+ 225us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:04.922995 (+ 299us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:04.923268 (+ 273us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:04.925661 (+ 2393us) write_op.cc:312] APPLY: finished
0810 23:33:04.926312 (+ 651us) log.cc:842] Serialized 23 byte log entry
0810 23:33:04.927320 (+ 1008us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:04.927505 (+ 185us) write_op.cc:454] Released schema lock
0810 23:33:04.927575 (+ 70us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:04.928575 (+ 1000us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:04.928606 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":18,"spinlock_wait_cycles":41409408,"child_traces":[["op",{"apply.queue_time_us":304,"apply.run_cpu_time_us":3379,"apply.run_wall_time_us":6751,"num_ops":1,"prepare.queue_time_us":422826,"prepare.run_cpu_time_us":2328,"prepare.run_wall_time_us":194465,"replication_time_us":304613}]]}
I20260810 23:33:07.705175 12224 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_8a358dec4da84beda7fdf1eb485eb587 [id=b5cc65e503f7413fbade0c94e2e74b14] alter complete (version 5)
I20260810 23:33:07.728662 13213 leader_election.cc:302] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:33:07.729900 15764 raft_consensus.cc:2781] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:07.730285 15764 raft_consensus.cc:482] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:07.730837 15764 raft_consensus.cc:3037] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:07.724886 13691 consensus_queue.cc:260] T 975a9bab969e46d2b939fc76ce350b99 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:33:07.740610 13213 leader_election.cc:341] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 1b1f24e226b94ccf8ad5a656c80a5b3b
I20260810 23:33:07.743201 13691 ts_tablet_manager.cc:1428] T 975a9bab969e46d2b939fc76ce350b99 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.081s user 0.004s sys 0.004s
I20260810 23:33:07.744720 13213 leader_election.cc:302] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
W20260810 23:33:07.745007 12240 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:56187 (request call id 70) took 829 ms (client timeout 1000 ms). Trace:
I20260810 23:33:07.745683 15744 raft_consensus.cc:2781] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:07.746233 15744 raft_consensus.cc:482] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:07.743636 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f53aca31ee4e338d9b5e281557c121" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:07.746726 15744 raft_consensus.cc:3037] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:07.749657 15796 tablet.cc:1722] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260810 23:33:07.758970 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdec13ee70d402fa3a08119e7bf1313 with delete_type TABLET_DATA_DELETED (replaced by tablet d603569c54ef4ffba9941d0cc3e0f97f) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:07.760727 15486 ts_tablet_manager.cc:1918] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:07.768675 15486 log.cc:1196] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/1b1f24e226b94ccf8ad5a656c80a5b3b
I20260810 23:33:07.769170 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c53785fdd24df79f138df6176e75d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:01 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:07.769634 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 806fe5162aef4d36ad5e2ac324047426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:07.770562 15486 ts_tablet_manager.cc:1939] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:07.771971 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 328) took 4093 ms. Trace:
W20260810 23:33:07.784966 12698 leader_election.cc:341] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:07.787429 13017 tablet_service.cc:1514] Processing DeleteTablet for tablet ab8d987109ba40198645881b449cb972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:02 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:07.790266 13691 tablet_bootstrap.cc:492] T df3ac117f3b34c1bb55b6a258d4fd539 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:07.790215 13017 tablet_service.cc:1514] Processing DeleteTablet for tablet a41ddf9f6f71443ba75c97d2b010372c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:07.793987 13017 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3049bcdc6e43c795961c74f0d008f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:41 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:07.794883 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e8311c941b43058384cee6d8263dc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:07.795732 12760 tablet_service.cc:1514] Processing DeleteTablet for tablet 422b25457d1549e186641fcedd8feab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:00 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:07.801076 15486 rpcz_store.cc:276] 0810 23:33:03.678910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.679303 (+ 393us) service_pool.cc:227] Handling call
0810 23:33:07.391946 (+3712643us) ts_tablet_manager.cc:1145] Deleting tablet 1b1f24e226b94ccf8ad5a656c80a5b3b
0810 23:33:07.516330 (+124384us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:07.658295 (+141965us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:07.760411 (+102116us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:07.771893 (+ 11482us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":151,"tablet-delete.queue_time_us":3684161}
I20260810 23:33:07.802881 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 336) took 3615 ms. Trace:
I20260810 23:33:07.803299 15486 rpcz_store.cc:276] 0810 23:33:04.187668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.187807 (+ 139us) service_pool.cc:227] Handling call
0810 23:33:07.802424 (+3614617us) ts_tablet_manager.cc:1145] Deleting tablet 95622c5782e443b9903c85f64852eaa9
0810 23:33:07.802481 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:07.802808 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3613851}
I20260810 23:33:07.804623 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 341) took 2394 ms. Trace:
I20260810 23:33:07.815816 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d0d9227d8574de8bd73041f3e8043a0 with delete_type TABLET_DATA_DELETED (replaced by tablet fb8d482c6ee54e45a41201d98a5b60b4) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:07.834030 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet f935119e4d9b4accb79db98185c80aca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:58 UTC) from {username='slave'} at 127.0.0.1:58598
W20260810 23:33:07.774605 13213 leader_election.cc:341] T 1b1f24e226b94ccf8ad5a656c80a5b3b P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Not found: Tablet not found: 1b1f24e226b94ccf8ad5a656c80a5b3b
W20260810 23:33:07.836894 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 62319.9us
I20260810 23:33:07.838413 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 62705f99687340aaa1aca0942b28487f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:48 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:07.847013 13213 leader_election.cc:302] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
W20260810 23:33:07.854029 12240 rpcz_store.cc:269] 0810 23:33:06.915563 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.055745 (+140182us) service_pool.cc:227] Handling call
0810 23:33:07.744899 (+689154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8096,"reported_tablets":22}
I20260810 23:33:07.808609 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 1b1f24e226b94ccf8ad5a656c80a5b3b (table default.Table_d942bf134f9849718bb9eb12c24806a1 [id=ff8088bdfc9f456683dd802d62ae67a6]) successfully deleted
I20260810 23:33:07.854769 13059 raft_consensus.cc:3037] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:33:07.855149 12214 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 46575.7us
I20260810 23:33:07.855427 13213 leader_election.cc:302] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
W20260810 23:33:07.856509 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 95622c5782e443b9903c85f64852eaa9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95622c5782e443b9903c85f64852eaa9 already in progress: creating tablet
I20260810 23:33:07.857190 15586 raft_consensus.cc:2726] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:33:07.858217 15586 raft_consensus.cc:2781] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:07.858803 15586 raft_consensus.cc:482] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:07.859382 15586 raft_consensus.cc:3037] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:07.869150 13213 leader_election.cc:302] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:33:07.832006 13665 tablet_bootstrap.cc:492] T ba8d56858471421a9fbc361ee6aff28f P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:07.870262 13665 ts_tablet_manager.cc:1397] T ba8d56858471421a9fbc361ee6aff28f P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.357s user 0.028s sys 0.011s
I20260810 23:33:07.781478 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet ab8d987109ba40198645881b449cb972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:02 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:07.873948 15486 rpcz_store.cc:276] 0810 23:33:05.410265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:05.410406 (+ 141us) service_pool.cc:227] Handling call
0810 23:33:07.804237 (+2393831us) ts_tablet_manager.cc:1145] Deleting tablet d53029158b0142078d66bad842f0a69e
0810 23:33:07.804282 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:07.804557 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393195}
I20260810 23:33:07.875574 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 345) took 2385 ms. Trace:
W20260810 23:33:07.876242 12698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.131:33807, user_credentials={real_user=slave}} blocked reactor thread for 83501.9us
W20260810 23:33:07.876583 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet d53029158b0142078d66bad842f0a69e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d53029158b0142078d66bad842f0a69e already in progress: creating tablet
I20260810 23:33:07.876399 15486 rpcz_store.cc:276] 0810 23:33:05.490759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:05.491706 (+ 947us) service_pool.cc:227] Handling call
0810 23:33:07.875140 (+2383434us) ts_tablet_manager.cc:1145] Deleting tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2
0810 23:33:07.875191 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:07.875507 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2382324}
I20260810 23:33:07.877784 15486 tablet_replica.cc:330] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
W20260810 23:33:07.877765 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2 already in progress: creating tablet
I20260810 23:33:07.880861 13691 tablet_bootstrap.cc:654] T df3ac117f3b34c1bb55b6a258d4fd539 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:07.869786 15689 raft_consensus.cc:504] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:07.888497 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 10033af79e854103bc47a046d2219ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:58 UTC) from {username='slave'} at 127.0.0.1:58598
W20260810 23:33:07.889042 12768 consensus_meta.cc:287] T 77b4478f4bc34dabb6e7cbff282442a4 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent flushing consensus metadata: real 0.547s user 0.017s sys 0.005s
I20260810 23:33:07.894912 12768 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 349) took 1036 ms. Trace:
I20260810 23:33:07.897552 12768 rpcz_store.cc:276] 0810 23:33:06.858181 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.858335 (+ 154us) service_pool.cc:227] Handling call
0810 23:33:06.978423 (+120088us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:33:06.978585 (+ 162us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:33:07.341860 (+363275us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:07.894003 (+552143us) tablet_replica.cc:205] Creating consensus instance
0810 23:33:07.894828 (+ 825us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:33:07.894229 13514 raft_consensus.cc:348] T bd921e48f22348639f7549d47b165185 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.899129 13514 raft_consensus.cc:374] T bd921e48f22348639f7549d47b165185 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 23:33:07.837805 12223 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 47) took 1764 ms (client timeout 1000 ms). Trace:
W20260810 23:33:07.768563 12236 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 43) took 1403 ms (client timeout 1000 ms). Trace:
I20260810 23:33:07.899524 13514 raft_consensus.cc:729] T bd921e48f22348639f7549d47b165185 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:33:07.896929 15689 leader_election.cc:288] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Requested vote from peers f26517150f1c480a877f15aec71d020b (127.11.226.130:42079), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
W20260810 23:33:07.899834 12236 rpcz_store.cc:269] 0810 23:33:06.365581 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.380329 (+ 14748us) service_pool.cc:227] Handling call
0810 23:33:07.422736 (+1042407us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:07.422771 (+ 35us) write_op.cc:454] Released schema lock
0810 23:33:07.768409 (+345638us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:07.337016 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:07.337476 (+ 460us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:07.337505 (+ 29us) write_op.cc:435] Acquired schema lock
0810 23:33:07.337528 (+ 23us) tablet.cc:660] Decoding operations
0810 23:33:07.337892 (+ 364us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:07.337938 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:07.337962 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:07.338181 (+ 219us) tablet.cc:699] Row locks acquired
0810 23:33:07.338199 (+ 18us) write_op.cc:260] PREPARE: finished
0810 23:33:07.338357 (+ 158us) write_op.cc:270] Start()
0810 23:33:07.338844 (+ 487us) write_op.cc:276] Timestamp: P: 1786404787338288 usec, L: 0
0810 23:33:07.338869 (+ 25us) op_driver.cc:331] REPLICATION: starting
0810 23:33:07.339697 (+ 828us) log.cc:842] Serialized 356 byte log entry
0810 23:33:07.364395 (+ 24698us) op_driver.cc:444] REPLICATION: finished
0810 23:33:07.365018 (+ 623us) write_op.cc:301] APPLY: starting
0810 23:33:07.365142 (+ 124us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:07.365299 (+ 157us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:07.365533 (+ 234us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:07.366018 (+ 485us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:07.366273 (+ 255us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:07.366405 (+ 132us) write_op.cc:312] APPLY: finished
0810 23:33:07.366772 (+ 367us) log.cc:842] Serialized 23 byte log entry
0810 23:33:07.367092 (+ 320us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:07.367239 (+ 147us) write_op.cc:454] Released schema lock
0810 23:33:07.367303 (+ 64us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":58055,"reported_tablets":29,"spinlock_wait_cycles":3840,"child_traces":[["op",{"apply.queue_time_us":373,"apply.run_cpu_time_us":2622,"apply.run_wall_time_us":2622,"num_ops":1,"prepare.queue_time_us":4083,"prepare.run_cpu_time_us":3200,"prepare.run_wall_time_us":22695,"raft.queue_time_us":122,"raft.run_cpu_time_us":905,"raft.run_wall_time_us":906,"replication_time_us":25403}]]}
I20260810 23:33:07.900913 13514 consensus_queue.cc:260] T bd921e48f22348639f7549d47b165185 P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:33:07.907461 12223 rpcz_store.cc:269] 0810 23:33:06.073612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.295765 (+222153us) service_pool.cc:227] Handling call
0810 23:33:06.342512 (+ 46747us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:33:07.837698 (+1495186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57814,"reported_tablets":21,"spinlock_wait_cycles":92326144}
I20260810 23:33:07.909003 13514 ts_tablet_manager.cc:1428] T bd921e48f22348639f7549d47b165185 P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.200s user 0.003s sys 0.005s
I20260810 23:33:07.910163 13514 tablet_bootstrap.cc:492] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:33:07.912523 15486 raft_consensus.cc:2229] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:07.913909 13017 tablet_service.cc:1514] Processing DeleteTablet for tablet f47b8b57d5b44a018bc2d7b3f976d803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:07.914852 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d4c879ca6a402dbafe91940e7b367a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:39 UTC) from {username='slave'} at 127.0.0.1:58598
W20260810 23:33:07.914714 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 45673us
I20260810 23:33:07.915452 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8c0d1dacc344cbacb0c5988f0e2221" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:07.917089 13665 raft_consensus.cc:348] T ba8d56858471421a9fbc361ee6aff28f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.917960 13665 raft_consensus.cc:374] T ba8d56858471421a9fbc361ee6aff28f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:07.918210 13665 raft_consensus.cc:729] T ba8d56858471421a9fbc361ee6aff28f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:07.918823 13665 consensus_queue.cc:260] T ba8d56858471421a9fbc361ee6aff28f P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.920811 15486 raft_consensus.cc:2258] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:33:07.923197 12700 leader_election.cc:341] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:07.924248 12700 leader_election.cc:341] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:07.924620 12700 leader_election.cc:302] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:07.926806 13514 tablet_bootstrap.cc:654] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
W20260810 23:33:07.946964 12229 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 45) took 3970 ms (client timeout 1000 ms). Trace:
I20260810 23:33:07.960283 12770 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47546 (request call id 351) took 1035 ms. Trace:
I20260810 23:33:07.962378 15745 raft_consensus.cc:2781] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:07.962981 15745 raft_consensus.cc:482] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:07.963590 15745 raft_consensus.cc:3037] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:07.964370 15744 raft_consensus.cc:504] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:07.967370 15486 ts_tablet_manager.cc:1905] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:07.899011 15586 raft_consensus.cc:504] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
W20260810 23:33:07.979557 12229 rpcz_store.cc:269] 0810 23:33:03.975531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.975667 (+ 136us) service_pool.cc:227] Handling call
0810 23:33:04.013313 (+ 37646us) spinlock_profiling.cc:243] Waited 33.3 ms on lock 0x7b54000316d4. stack: 00007f0ce0390033 00007f0ce038ff30 00007f0cdff11534 00000000004db2b4 00000000004db249 00007f0ce729caf0 00007f0ce731ff2a 00007f0ce91ebd6c 00007f0ce92e5c3e 00007f0ce6e1cfb4 00007f0ce6e1cf42 00007f0ce6e1ce71 00007f0ce6e1cdec 00007f0ce6e1c0a2 00007f0ce50c694c 00007f0ce50c5d86
0810 23:33:07.946220 (+3932907us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23942,"reported_tablets":24,"spinlock_wait_cycles":76668672}
W20260810 23:33:07.952518 12224 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 46) took 2934 ms (client timeout 1000 ms). Trace:
I20260810 23:33:07.981925 13691 tablet_bootstrap.cc:492] T df3ac117f3b34c1bb55b6a258d4fd539 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:07.982424 13691 ts_tablet_manager.cc:1397] T df3ac117f3b34c1bb55b6a258d4fd539 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.199s user 0.015s sys 0.009s
W20260810 23:33:07.980990 12224 rpcz_store.cc:269] 0810 23:33:05.018494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:05.289781 (+271287us) service_pool.cc:227] Handling call
0810 23:33:07.952399 (+2662618us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:07.593285 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:07.593781 (+ 496us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:07.593810 (+ 29us) write_op.cc:435] Acquired schema lock
0810 23:33:07.593825 (+ 15us) tablet.cc:660] Decoding operations
0810 23:33:07.594292 (+ 467us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:07.594335 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:07.594357 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:07.594555 (+ 198us) tablet.cc:699] Row locks acquired
0810 23:33:07.594571 (+ 16us) write_op.cc:260] PREPARE: finished
0810 23:33:07.594727 (+ 156us) write_op.cc:270] Start()
0810 23:33:07.594822 (+ 95us) write_op.cc:276] Timestamp: P: 1786404787594643 usec, L: 0
0810 23:33:07.594852 (+ 30us) op_driver.cc:331] REPLICATION: starting
0810 23:33:07.595460 (+ 608us) log.cc:842] Serialized 356 byte log entry
0810 23:33:07.614605 (+ 19145us) op_driver.cc:444] REPLICATION: finished
0810 23:33:07.615004 (+ 399us) write_op.cc:301] APPLY: starting
0810 23:33:07.615092 (+ 88us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:07.615230 (+ 138us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:07.615440 (+ 210us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:07.615879 (+ 439us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:07.616126 (+ 247us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:07.616222 (+ 96us) write_op.cc:312] APPLY: finished
0810 23:33:07.616542 (+ 320us) log.cc:842] Serialized 23 byte log entry
0810 23:33:07.616792 (+ 250us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:07.616903 (+ 111us) write_op.cc:454] Released schema lock
0810 23:33:07.616963 (+ 60us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:07.617257 (+ 294us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:07.617299 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":94999,"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":172,"apply.run_cpu_time_us":2532,"apply.run_wall_time_us":2537,"num_ops":1,"prepare.queue_time_us":156,"prepare.run_cpu_time_us":2554,"prepare.run_wall_time_us":2554,"raft.queue_time_us":522,"raft.run_cpu_time_us":1364,"raft.run_wall_time_us":1364,"replication_time_us":19638}]]}
I20260810 23:33:07.992689 15586 leader_election.cc:288] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:07.993220 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:33:07.994223 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]). Expected version 1 got 0
I20260810 23:33:07.995067 12240 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet a6f53aca31ee4e338d9b5e281557c121 (table default.Table_7e1011c9a03343acb3bdf04b07eb962e [id=89c842ab64c546bbb967a2246a7220ac]). Expected version 3 got 0
I20260810 23:33:08.008426 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 240adfa4be844518bbc9624d0e408789 with delete_type TABLET_DATA_DELETED (replaced by tablet 4344987a6c90428fbdf50765a4631455) from {username='slave'} at 127.0.0.1:58598
W20260810 23:33:08.001556 12700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 77322.2us
I20260810 23:33:08.002251 15796 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:53280 (request call id 343) took 2517 ms. Trace:
I20260810 23:33:08.013298 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:51 UTC) from {username='slave'} at 127.0.0.1:58598
W20260810 23:33:08.016332 12235 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:47217 (request call id 48) took 866 ms (client timeout 1000 ms). Trace:
I20260810 23:33:08.019044 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8c0d1dacc344cbacb0c5988f0e2221" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
W20260810 23:33:08.019726 12235 rpcz_store.cc:269] 0810 23:33:07.150309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.231666 (+ 81357us) service_pool.cc:227] Handling call
0810 23:33:08.016209 (+784543us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50507,"reported_tablets":22}
W20260810 23:33:08.021502 13212 leader_election.cc:341] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:33:08.027010 13212 consensus_peers.cc:487] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Couldn't send request to peer f26517150f1c480a877f15aec71d020b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:08.027551 13691 raft_consensus.cc:348] T df3ac117f3b34c1bb55b6a258d4fd539 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.028574 13691 raft_consensus.cc:374] T df3ac117f3b34c1bb55b6a258d4fd539 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:08.028891 13691 raft_consensus.cc:729] T df3ac117f3b34c1bb55b6a258d4fd539 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:08.029695 15744 leader_election.cc:288] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:08.029803 13691 consensus_queue.cc:260] T df3ac117f3b34c1bb55b6a258d4fd539 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.031630 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975a9bab969e46d2b939fc76ce350b99" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:08.032353 13057 raft_consensus.cc:3037] T 975a9bab969e46d2b939fc76ce350b99 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:07.961635 12770 rpcz_store.cc:276] 0810 23:33:06.924771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.924957 (+ 186us) service_pool.cc:227] Handling call
0810 23:33:06.981735 (+ 56778us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0810 23:33:06.981842 (+ 107us) ts_tablet_manager.cc:676] Creating new metadata...
0810 23:33:07.504530 (+522688us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:07.871651 (+367121us) tablet_replica.cc:205] Creating consensus instance
0810 23:33:07.960192 (+ 88541us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260810 23:33:08.039009 15687 raft_consensus.cc:2726] T a6f53aca31ee4e338d9b5e281557c121 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:33:08.041594 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 125918us
I20260810 23:33:08.042572 13213 leader_election.cc:302] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
W20260810 23:33:08.043862 12700 leader_election.cc:341] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:08.050469 15807 raft_consensus.cc:2781] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:08.051288 15807 raft_consensus.cc:482] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:08.051908 15807 raft_consensus.cc:3037] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:33:08.056855 13213 leader_election.cc:341] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:08.057374 13213 leader_election.cc:302] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:08.011543 15796 rpcz_store.cc:276] 0810 23:33:05.485253 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:05.485488 (+ 235us) service_pool.cc:227] Handling call
0810 23:33:08.002048 (+2516560us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:05.566064 (+ 0us) alter_schema_op.cc:102] PREPARE ALTER-SCHEMA: Starting
0810 23:33:05.566663 (+ 599us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0810 23:33:05.566711 (+ 48us) alter_schema_op.cc:65] Acquired schema lock
0810 23:33:05.568981 (+ 2270us) alter_schema_op.cc:117] PREPARE ALTER-SCHEMA: finished
0810 23:33:05.569844 (+ 863us) alter_schema_op.cc:126] START. Timestamp: 1786404785569661
0810 23:33:05.569884 (+ 40us) op_driver.cc:331] REPLICATION: starting
0810 23:33:05.570715 (+ 831us) log.cc:842] Serialized 173 byte log entry
0810 23:33:07.668383 (+2097668us) op_driver.cc:444] REPLICATION: finished
0810 23:33:07.749511 (+ 81128us) alter_schema_op.cc:131] APPLY ALTER-SCHEMA: Starting
0810 23:33:07.763104 (+ 13593us) log.cc:842] Serialized 5 byte log entry
0810 23:33:07.997601 (+234497us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:07.999218 (+ 1617us) log.cc:842] Serialized 14 byte log entry
0810 23:33:08.001721 (+ 2503us) alter_schema_op.cc:71] Released schema lock
0810 23:33:08.001778 (+ 57us) alter_schema_op.cc:184] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":80832,"mutex_wait_us":1197,"prepare.queue_time_us":72319,"prepare.run_cpu_time_us":19575,"prepare.run_wall_time_us":98499,"raft.queue_time_us":37361,"raft.run_cpu_time_us":1306,"raft.run_wall_time_us":1899,"replication_time_us":2098394,"spinlock_wait_cycles":1363712,"thread_start_us":328985,"threads_started":4,"wal-append.queue_time_us":122372,"wal-append.run_cpu_time_us":6159,"wal-append.run_wall_time_us":1048089}]]}
I20260810 23:33:08.062546 13665 ts_tablet_manager.cc:1428] T ba8d56858471421a9fbc361ee6aff28f P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.192s user 0.004s sys 0.011s
I20260810 23:33:08.066236 13514 tablet_bootstrap.cc:492] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3: No bootstrap required, opened a new log
I20260810 23:33:08.066792 13514 ts_tablet_manager.cc:1397] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3: Time spent bootstrapping tablet: real 0.157s user 0.005s sys 0.022s
I20260810 23:33:08.069370 13691 ts_tablet_manager.cc:1428] T df3ac117f3b34c1bb55b6a258d4fd539 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.087s user 0.008s sys 0.004s
I20260810 23:33:08.070421 15455 raft_consensus.cc:482] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:08.076502 15455 raft_consensus.cc:504] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.078629 15586 raft_consensus.cc:482] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:08.078845 15777 raft_consensus.cc:482] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:08.079370 15586 raft_consensus.cc:504] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.081491 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806fe5162aef4d36ad5e2ac324047426" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:08.079402 15777 raft_consensus.cc:504] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.083791 15812 raft_consensus.cc:2726] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:08.084697 15777 leader_election.cc:288] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:08.084987 15455 leader_election.cc:288] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:08.072954 14121 ts_tablet_manager.cc:1905] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:08.089803 12794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806fe5162aef4d36ad5e2ac324047426" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:08.092201 15082 ts_tablet_manager.cc:1918] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260810 23:33:08.092655 15082 log.cc:1196] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/88c53785fdd24df79f138df6176e75d2
I20260810 23:33:08.093504 15082 ts_tablet_manager.cc:1939] T 88c53785fdd24df79f138df6176e75d2 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:33:08.090925 12794 raft_consensus.cc:2454] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 0.
I20260810 23:33:08.098430 12958 leader_election.cc:302] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters:
I20260810 23:33:08.099524 15455 raft_consensus.cc:2781] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:08.100172 15455 raft_consensus.cc:482] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:08.100924 15455 raft_consensus.cc:3037] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:08.101258 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b54edbb45d496c94efedb6feee8e45" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:33:08.103374 12700 leader_election.cc:341] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:08.111474 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b54edbb45d496c94efedb6feee8e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:08.117038 13691 tablet_bootstrap.cc:492] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:08.120890 15816 tablet.cc:1722] T ff7bd9b6ca4845d8b8705ff970c8f06e P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260810 23:33:08.129609 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 72793us
W20260810 23:33:08.133149 13213 leader_election.cc:341] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:08.133644 13213 leader_election.cc:302] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:08.135784 13665 tablet_bootstrap.cc:492] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:08.106536 12234 catalog_manager.cc:5495] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } }
I20260810 23:33:08.142654 12234 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]). Expected version 1 got 0
I20260810 23:33:08.143635 13665 tablet_bootstrap.cc:654] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:08.144460 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 306) took 6496 ms. Trace:
I20260810 23:33:08.144819 15082 rpcz_store.cc:276] 0810 23:33:01.648159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:01.648508 (+ 349us) service_pool.cc:227] Handling call
0810 23:33:07.207742 (+5559234us) ts_tablet_manager.cc:1145] Deleting tablet 88c53785fdd24df79f138df6176e75d2
0810 23:33:07.213650 (+ 5908us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:07.289089 (+ 75439us) spinlock_profiling.cc:243] Waited 7.84 ms on lock 0x7b54000a19d4. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3c5352 00007f3cde7db26c 00007f3cdf3e8cfa 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49 00007f3cdf3ffd11 00007f3cdf3fef3d 0000000000580784
0810 23:33:07.778225 (+489136us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.091867 (+313642us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.144383 (+ 52516us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18036608,"tablet-delete.queue_time_us":5553833}
I20260810 23:33:08.145895 12234 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet ff7bd9b6ca4845d8b8705ff970c8f06e (table default.Table_4a7751c5b61c4332adeeb9ee00474b27 [id=2c8320255aab4036b5350d082f9e04ef]). Expected version 2 got 0
I20260810 23:33:08.146507 15745 raft_consensus.cc:504] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.148011 13327 raft_consensus.cc:2379] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:33:08.147895 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet 88c53785fdd24df79f138df6176e75d2 (table default.Table_d0b8422dca5e40ff919741fcee7d547c [id=71c9424f327040a68889fc257935c535]) successfully deleted
I20260810 23:33:08.150318 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c6a6e6fb6124c4bbdae70eb444f6fdd" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:08.150847 12234 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:33:08.149384 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 307) took 6275 ms. Trace:
W20260810 23:33:08.152206 13212 leader_election.cc:341] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:08.153139 15082 rpcz_store.cc:276] 0810 23:33:01.874309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:01.874462 (+ 153us) service_pool.cc:227] Handling call
0810 23:33:08.148953 (+6274491us) ts_tablet_manager.cc:1145] Deleting tablet 8163921fdc2c442cb335041c79de6d6d
0810 23:33:08.149012 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.149324 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6271356}
I20260810 23:33:08.156574 15586 leader_election.cc:288] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:08.157827 13514 raft_consensus.cc:348] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.158831 13514 raft_consensus.cc:374] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:08.163095 15786 tablet.cc:1722] T ff7bd9b6ca4845d8b8705ff970c8f06e P f26517150f1c480a877f15aec71d020b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260810 23:33:08.155212 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 8163921fdc2c442cb335041c79de6d6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8163921fdc2c442cb335041c79de6d6d already in progress: creating tablet
I20260810 23:33:08.166460 13057 raft_consensus.cc:2454] T 975a9bab969e46d2b939fc76ce350b99 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:33:08.166070 15764 raft_consensus.cc:504] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.166793 15777 raft_consensus.cc:482] T bd921e48f22348639f7549d47b165185 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:08.167339 15777 raft_consensus.cc:504] T bd921e48f22348639f7549d47b165185 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.168497 15764 leader_election.cc:288] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:08.168743 15821 tablet.cc:1722] T c5794637d5dd43f2b05edd53865f4e99 P 191d120f4c7f44fc8475305a6d2e75f3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 23:33:08.170303 15777 leader_election.cc:288] T bd921e48f22348639f7549d47b165185 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:08.171161 15486 ts_tablet_manager.cc:1918] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:33:08.171581 15486 log.cc:1196] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/8fb2ee1c9b41478da001aa67d7a7ad3c
I20260810 23:33:08.172183 15744 raft_consensus.cc:2726] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:08.174773 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3ac117f3b34c1bb55b6a258d4fd539" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:08.175392 13058 raft_consensus.cc:3037] T df3ac117f3b34c1bb55b6a258d4fd539 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:08.180603 13212 leader_election.cc:302] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:08.181617 13691 tablet_bootstrap.cc:654] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:08.186741 13058 raft_consensus.cc:2454] T df3ac117f3b34c1bb55b6a258d4fd539 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:33:08.188087 15744 raft_consensus.cc:2781] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:08.188680 15744 raft_consensus.cc:686] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:33:08.188301 15807 raft_consensus.cc:504] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.189762 15744 consensus_queue.cc:237] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.191076 15807 leader_election.cc:288] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:08.196750 12745 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:47536 (request call id 85) took 1550 ms. Trace:
W20260810 23:33:08.197018 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 63910.5us
I20260810 23:33:08.197201 12745 rpcz_store.cc:276] 0810 23:33:06.646136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.723663 (+ 77527us) service_pool.cc:227] Handling call
0810 23:33:07.467367 (+743704us) spinlock_profiling.cc:243] Waited 37.5 ms on lock 0x7b5c00042b8c. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde752361 00007f3cde8ebfa9 00007f3cde7de22a 00007f3cdf3a36a0 00007f3cddc4be04 00007f3cddc4bd92 00007f3cddc4bcc1 00007f3cddc4bc3c 00007f3cddc4aef2 00007f3cdc16c94c 00007f3cdc16bd86
0810 23:33:08.127075 (+659708us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b5400130bd4. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cde342471 00007f3cde3b7268 00007f3cdf3a36cd 00007f3cddc4be04 00007f3cddc4bd92 00007f3cddc4bcc1 00007f3cddc4bc3c 00007f3cddc4aef2 00007f3cdc16c94c 00007f3cdc16bd86 00007f3cdc16b72f
0810 23:33:08.196653 (+ 69578us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":665268352}
I20260810 23:33:08.159138 13514 raft_consensus.cc:729] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191d120f4c7f44fc8475305a6d2e75f3, State: Initialized, Role: FOLLOWER
I20260810 23:33:08.156263 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 309) took 5941 ms. Trace:
I20260810 23:33:08.200079 15082 rpcz_store.cc:276] 0810 23:33:02.215301 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:02.215483 (+ 182us) service_pool.cc:227] Handling call
0810 23:33:08.155787 (+5940304us) ts_tablet_manager.cc:1145] Deleting tablet 40eb874e0b334c46961e16d8d5aaaaa7
0810 23:33:08.155841 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.156207 (+ 366us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5939567}
W20260810 23:33:08.202028 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 40eb874e0b334c46961e16d8d5aaaaa7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40eb874e0b334c46961e16d8d5aaaaa7 already in progress: creating tablet
I20260810 23:33:08.206678 12234 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]). Expected version 1 got 0
I20260810 23:33:08.208647 15486 ts_tablet_manager.cc:1939] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:08.214769 13514 consensus_queue.cc:260] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.216996 15745 leader_election.cc:288] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:08.221673 12234 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet c5794637d5dd43f2b05edd53865f4e99 (table default.Table_cf212e8521974a72a6b86f2ea6ef2ffc [id=339a844c37ee460cb948d7e58264e5c8]). Expected version 1 got 0
I20260810 23:33:08.222498 12234 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 00f68708a0b74a58b69581e8324a33f6 (table default.Table_d134a55028664442b08b229593877973 [id=26be70827ca442a4b24a26015ffd952a]). Expected version 2 got 0
I20260810 23:33:08.207913 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd921e48f22348639f7549d47b165185" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:08.216413 15455 raft_consensus.cc:504] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:33:08.230561 12700 leader_election.cc:341] T bd921e48f22348639f7549d47b165185 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:08.231419 13514 ts_tablet_manager.cc:1428] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3: Time spent starting tablet: real 0.164s user 0.008s sys 0.002s
I20260810 23:33:08.232620 13514 tablet_bootstrap.cc:492] T e62dc27b59ba469d99e8d11372aceff5 P 191d120f4c7f44fc8475305a6d2e75f3: Bootstrap starting.
I20260810 23:33:08.234876 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806fe5162aef4d36ad5e2ac324047426" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:08.235654 12806 raft_consensus.cc:3037] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:33:08.225600 13213 consensus_peers.cc:487] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a480093c481941f0bb35949c7606f37f. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:08.229943 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 344) took 2741 ms. Trace:
I20260810 23:33:08.233999 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806fe5162aef4d36ad5e2ac324047426" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:33:08.238651 13327 raft_consensus.cc:2379] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:33:08.215862 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 310) took 5961 ms. Trace:
W20260810 23:33:08.237301 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 56800.3us
I20260810 23:33:08.240314 15082 rpcz_store.cc:276] 0810 23:33:02.254190 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:02.254397 (+ 207us) service_pool.cc:227] Handling call
0810 23:33:08.203458 (+5949061us) ts_tablet_manager.cc:1145] Deleting tablet 7031c257fdf34a28b6954d3d4827a8da
0810 23:33:08.215311 (+ 11853us) spinlock_profiling.cc:243] Waited 9.85 ms on lock 0x7b5800001898. stack: 00007f3cd7436033 00007f3cd7435f30 00007f3cd6fb7534 00000000004db2b4 00000000004db249 00007f3cd72acf11 00007f3cd7430851 00007f3cd7430e82 00007f3ce02f08b9 00007f3cdf3245d8 00007f3cdf3e8389 00007f3cdf3e8983 00007f3cdf418e10 00007f3cdf3ffe0d 00007f3cdf3ffdb9 00007f3cdf3ffd49
0810 23:33:08.215372 (+ 61us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.215797 (+ 425us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22659328,"tablet-delete.queue_time_us":5948229}
I20260810 23:33:08.241670 12797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3ac117f3b34c1bb55b6a258d4fd539" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
W20260810 23:33:08.242062 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet 7031c257fdf34a28b6954d3d4827a8da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7031c257fdf34a28b6954d3d4827a8da already in progress: creating tablet
I20260810 23:33:08.242456 12797 raft_consensus.cc:3037] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:08.242311 12794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8c0d1dacc344cbacb0c5988f0e2221" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:08.243259 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f68708a0b74a58b69581e8324a33f6" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:08.243547 12794 raft_consensus.cc:2379] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in current term 1: Already voted for candidate 191d120f4c7f44fc8475305a6d2e75f3 in this term.
I20260810 23:33:08.241748 12802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f68708a0b74a58b69581e8324a33f6" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:08.244374 13212 leader_election.cc:302] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters:
I20260810 23:33:08.244565 12802 raft_consensus.cc:3037] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:08.244736 15455 leader_election.cc:288] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Requested vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:08.244650 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975a9bab969e46d2b939fc76ce350b99" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:08.245306 12805 raft_consensus.cc:3037] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:08.245422 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806fe5162aef4d36ad5e2ac324047426" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b"
I20260810 23:33:08.246340 13058 raft_consensus.cc:2379] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in current term 1: Already voted for candidate f26517150f1c480a877f15aec71d020b in this term.
I20260810 23:33:08.247272 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 311) took 5911 ms. Trace:
I20260810 23:33:08.247923 12803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c6a6e6fb6124c4bbdae70eb444f6fdd" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:08.247720 15082 rpcz_store.cc:276] 0810 23:33:02.335564 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:02.335735 (+ 171us) service_pool.cc:227] Handling call
0810 23:33:08.246529 (+5910794us) ts_tablet_manager.cc:1145] Deleting tablet dfda619f19a5459494b1e4f6d22821cc
0810 23:33:08.246653 (+ 124us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.247187 (+ 534us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5910060}
W20260810 23:33:08.253028 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet dfda619f19a5459494b1e4f6d22821cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfda619f19a5459494b1e4f6d22821cc already in progress: creating tablet
I20260810 23:33:08.254714 15777 raft_consensus.cc:482] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:08.254586 12800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806fe5162aef4d36ad5e2ac324047426" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3"
I20260810 23:33:08.255232 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 324) took 5109 ms. Trace:
I20260810 23:33:08.255414 15777 raft_consensus.cc:504] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.255731 15082 rpcz_store.cc:276] 0810 23:33:03.145774 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.194660 (+ 48886us) service_pool.cc:227] Handling call
0810 23:33:08.254379 (+5059719us) ts_tablet_manager.cc:1145] Deleting tablet f47d8e462d814a15a82fd0d5218e2d0e
0810 23:33:08.254475 (+ 96us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.255107 (+ 632us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5058418}
W20260810 23:33:08.257447 12217 catalog_manager.cc:4821] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): delete failed for tablet f47d8e462d814a15a82fd0d5218e2d0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f47d8e462d814a15a82fd0d5218e2d0e already in progress: creating tablet
I20260810 23:33:08.258338 15777 leader_election.cc:288] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:08.259713 15082 tablet_replica.cc:330] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:33:08.263890 15082 raft_consensus.cc:2229] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:08.266594 15082 raft_consensus.cc:2258] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:08.267024 13691 tablet_bootstrap.cc:492] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:08.267553 13691 ts_tablet_manager.cc:1397] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.151s user 0.008s sys 0.004s
I20260810 23:33:08.273195 12797 raft_consensus.cc:2454] T df3ac117f3b34c1bb55b6a258d4fd539 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:33:08.240178 15486 rpcz_store.cc:276] 0810 23:33:05.488481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:05.488764 (+ 283us) service_pool.cc:227] Handling call
0810 23:33:07.877610 (+2388846us) ts_tablet_manager.cc:1145] Deleting tablet 8fb2ee1c9b41478da001aa67d7a7ad3c
0810 23:33:07.877656 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.025399 (+147743us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.170869 (+145470us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.229855 (+ 58986us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25038,"tablet-delete.queue_time_us":2374663}
I20260810 23:33:08.294821 13665 tablet_bootstrap.cc:492] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:08.295527 13665 ts_tablet_manager.cc:1397] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.160s user 0.014s sys 0.000s
I20260810 23:33:08.301005 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8d56858471421a9fbc361ee6aff28f" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
I20260810 23:33:08.303762 15813 consensus_queue.cc:1035] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:33:08.308265 13212 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42079, user_credentials={real_user=slave}} blocked reactor thread for 63995.5us
W20260810 23:33:08.309414 13212 leader_election.cc:341] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:08.309338 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5]) successfully deleted
I20260810 23:33:08.310166 14121 ts_tablet_manager.cc:1918] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:33:08.275840 15082 ts_tablet_manager.cc:1905] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:08.278447 12802 raft_consensus.cc:2454] T 00f68708a0b74a58b69581e8324a33f6 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:33:08.310971 15812 raft_consensus.cc:2781] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:08.311808 15812 raft_consensus.cc:686] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:33:08.313469 15812 consensus_queue.cc:237] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:33:08.303810 12700 leader_election.cc:341] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:08.315564 14121 log.cc:1196] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/8fb2ee1c9b41478da001aa67d7a7ad3c
I20260810 23:33:08.287462 13691 raft_consensus.cc:348] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:08.320776 13691 raft_consensus.cc:374] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:08.321164 13691 raft_consensus.cc:729] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:08.322113 13691 consensus_queue.cc:260] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:08.334645 12805 raft_consensus.cc:2454] T 975a9bab969e46d2b939fc76ce350b99 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in term 1.
I20260810 23:33:08.337677 13691 ts_tablet_manager.cc:1428] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.070s user 0.008s sys 0.002s
I20260810 23:33:08.320451 14121 ts_tablet_manager.cc:1939] T 8fb2ee1c9b41478da001aa67d7a7ad3c P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:33:08.340508 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 331) took 2962 ms. Trace:
I20260810 23:33:08.340981 14121 rpcz_store.cc:276] 0810 23:33:05.378142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:05.378318 (+ 176us) service_pool.cc:227] Handling call
0810 23:33:07.291584 (+1913266us) ts_tablet_manager.cc:1145] Deleting tablet 8fb2ee1c9b41478da001aa67d7a7ad3c
0810 23:33:07.291635 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:07.410166 (+118531us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b5400092c54. stack: 00007fa86ed56033 00007fa86ed55f30 00007fa86e8d7534 00000000004db2b4 00000000004db249 00007fa875c62471 00007fa875ce5352 00007fa8760fb26c 00007fa876d08cfa 00007fa876d38e10 00007fa876d1fe0d 00007fa876d1fdb9 00007fa876d1fd49 00007fa876d1fd11 00007fa876d1ef3d 0000000000580784
0810 23:33:08.147196 (+737030us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.309846 (+162650us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.340433 (+ 30587us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":265906944,"tablet-delete.queue_time_us":1912598}
I20260810 23:33:08.343091 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet 8fb2ee1c9b41478da001aa67d7a7ad3c (table default.Table_1476071cacef4a1b8681d049b2d23db5 [id=dae244793fd24816a8199c38f23a25b5]) successfully deleted
I20260810 23:33:08.344870 13691 tablet_bootstrap.cc:492] T 00f68708a0b74a58b69581e8324a33f6 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:08.323369 15486 tablet_replica.cc:330] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:08.346828 12240 master_service.cc:946] ReplaceTablet: received request to replace tablet 0e8120281fe84e9a89f25d07c1d7b6a8 from {username='slave'} at 127.0.0.1:34742
I20260810 23:33:08.347690 15486 raft_consensus.cc:2229] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:08.345737 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 332) took 2913 ms. Trace:
I20260810 23:33:08.355798 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8c0d1dacc344cbacb0c5988f0e2221" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3"
I20260810 23:33:08.356889 13336 raft_consensus.cc:2379] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 191d120f4c7f44fc8475305a6d2e75f3 in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:33:08.359666 15486 raft_consensus.cc:2258] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:08.361513 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8d56858471421a9fbc361ee6aff28f" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:08.362587 13335 raft_consensus.cc:2454] T ba8d56858471421a9fbc361ee6aff28f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:33:08.363647 13665 raft_consensus.cc:348] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.364418 13665 raft_consensus.cc:374] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:08.366904 12698 leader_election.cc:302] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:08.361861 12806 raft_consensus.cc:2454] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26517150f1c480a877f15aec71d020b in term 1.
I20260810 23:33:08.368340 15410 raft_consensus.cc:2726] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:33:08.370273 12958 leader_election.cc:302] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b; no voters: f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:08.352061 14121 rpcz_store.cc:276] 0810 23:33:05.432727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:05.432906 (+ 179us) service_pool.cc:227] Handling call
0810 23:33:08.345317 (+2912411us) ts_tablet_manager.cc:1145] Deleting tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2
0810 23:33:08.345384 (+ 67us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.345675 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2911616}
W20260810 23:33:08.372486 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ecf2c2f86db4052b79a8ce2c9e6a4a2 already in progress: creating tablet
I20260810 23:33:08.374249 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd921e48f22348639f7549d47b165185" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:08.375339 13327 raft_consensus.cc:2454] T bd921e48f22348639f7549d47b165185 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191d120f4c7f44fc8475305a6d2e75f3 in term 0.
I20260810 23:33:08.376303 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b54edbb45d496c94efedb6feee8e45" candidate_uuid: "191d120f4c7f44fc8475305a6d2e75f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:08.364694 13665 raft_consensus.cc:729] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:08.379843 15486 ts_tablet_manager.cc:1905] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:08.380908 13665 consensus_queue.cc:260] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:33:08.386402 12698 consensus_peers.cc:487] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 -> Peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Couldn't send request to peer f54aa78a60784a3bb2f5eb4bb2401ce3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:08.392197 13213 leader_election.cc:341] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:08.397712 14121 tablet_replica.cc:330] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:33:08.402886 14121 raft_consensus.cc:2229] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:08.403612 13213 leader_election.cc:302] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:08.405778 13514 tablet_bootstrap.cc:654] T e62dc27b59ba469d99e8d11372aceff5 P 191d120f4c7f44fc8475305a6d2e75f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:08.400693 13665 ts_tablet_manager.cc:1428] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.105s user 0.009s sys 0.000s
I20260810 23:33:08.411116 15764 raft_consensus.cc:2726] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:08.411245 15455 raft_consensus.cc:2781] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:08.412878 15700 raft_consensus.cc:482] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:08.413060 13213 leader_election.cc:302] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:08.413558 15700 raft_consensus.cc:504] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:08.423560 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8c0d1dacc344cbacb0c5988f0e2221" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:08.425442 13327 raft_consensus.cc:2379] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:33:08.426352 15082 ts_tablet_manager.cc:1918] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:33:08.426882 15082 log.cc:1196] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/ab8d987109ba40198645881b449cb972
I20260810 23:33:08.427663 15082 ts_tablet_manager.cc:1939] T ab8d987109ba40198645881b449cb972 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting consensus metadata
I20260810 23:33:08.434862 15700 leader_election.cc:288] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:33:08.409250 13665 tablet_bootstrap.cc:492] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:08.449234 14121 raft_consensus.cc:2258] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:08.452593 15764 raft_consensus.cc:2726] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260810 23:33:08.455420 15664 lock_manager.cc:586] Waited 3 seconds to obtain row lock on key '<redacted>' (key hash 8606919409353307012) tablet 00000000000000000000000000000000 cur holder 0x7b5800021600
I20260810 23:33:08.455415 13213 leader_election.cc:302] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:08.462741 15082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47546 (request call id 325) took 4972 ms. Trace:
I20260810 23:33:08.463155 15082 rpcz_store.cc:276] 0810 23:33:03.490465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.490856 (+ 391us) service_pool.cc:227] Handling call
0810 23:33:08.259446 (+4768590us) ts_tablet_manager.cc:1145] Deleting tablet ab8d987109ba40198645881b449cb972
0810 23:33:08.259536 (+ 90us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.374845 (+115309us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.426068 (+ 51223us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.462631 (+ 36563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4767939}
I20260810 23:33:08.464486 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8c0d1dacc344cbacb0c5988f0e2221" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191d120f4c7f44fc8475305a6d2e75f3" is_pre_election: true
I20260810 23:33:08.464840 12217 catalog_manager.cc:4841] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): tablet ab8d987109ba40198645881b449cb972 (table default.Table_d8a0e3e655b9417d9df3ff6d82b76f15 [id=d0b86eafe9544cd98d8fce8ece3b5c10]) successfully deleted
I20260810 23:33:08.465333 12805 raft_consensus.cc:2379] T ca8c0d1dacc344cbacb0c5988f0e2221 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate 191d120f4c7f44fc8475305a6d2e75f3 in this term.
I20260810 23:33:08.466815 12958 leader_election.cc:302] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f26517150f1c480a877f15aec71d020b; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:08.471277 15813 raft_consensus.cc:2726] T ca8c0d1dacc344cbacb0c5988f0e2221 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:33:08.473387 15836 raft_consensus.cc:3037] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:08.474916 13213 leader_election.cc:302] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
I20260810 23:33:08.475566 15455 raft_consensus.cc:686] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [term 1 LEADER]: Becoming Leader. State: Replica: f26517150f1c480a877f15aec71d020b, State: Running, Role: LEADER
I20260810 23:33:08.476691 15082 tablet_replica.cc:330] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3: stopping tablet replica
I20260810 23:33:08.477680 15082 raft_consensus.cc:2229] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:08.478266 15082 raft_consensus.cc:2258] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:33:08.485916 12700 consensus_peers.cc:487] T dce0ec83ed8943a0b0788505bb75d1df P 191d120f4c7f44fc8475305a6d2e75f3 -> Peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Couldn't send request to peer f26517150f1c480a877f15aec71d020b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:08.487913 15082 ts_tablet_manager.cc:1905] T 806fe5162aef4d36ad5e2ac324047426 P 191d120f4c7f44fc8475305a6d2e75f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:08.493817 15764 raft_consensus.cc:2781] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:08.494650 15764 raft_consensus.cc:686] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Becoming Leader. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Running, Role: LEADER
I20260810 23:33:08.495949 15764 consensus_queue.cc:237] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.500924 14121 ts_tablet_manager.cc:1905] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:08.477140 15455 consensus_queue.cc:237] T 806fe5162aef4d36ad5e2ac324047426 P f26517150f1c480a877f15aec71d020b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.533123 12229 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:33:08.534143 12229 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]). Expected version 1 got 0
I20260810 23:33:08.545243 12235 catalog_manager.cc:5587] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:33:08.546330 13691 tablet_bootstrap.cc:654] T 00f68708a0b74a58b69581e8324a33f6 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:08.554623 12698 leader_election.cc:302] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3, f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: f26517150f1c480a877f15aec71d020b
W20260810 23:33:08.556934 12698 leader_election.cc:341] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:08.557539 12698 leader_election.cc:302] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191d120f4c7f44fc8475305a6d2e75f3; no voters: f26517150f1c480a877f15aec71d020b, f54aa78a60784a3bb2f5eb4bb2401ce3
I20260810 23:33:08.558615 15777 raft_consensus.cc:2781] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:08.558665 15410 raft_consensus.cc:2726] T 80b54edbb45d496c94efedb6feee8e45 P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:08.559216 15777 raft_consensus.cc:482] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:08.559767 15777 raft_consensus.cc:3037] T ba8d56858471421a9fbc361ee6aff28f P 191d120f4c7f44fc8475305a6d2e75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:08.560884 12240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: Sending DeleteTablet for 3 replicas of tablet 0e8120281fe84e9a89f25d07c1d7b6a8
I20260810 23:33:08.562605 12240 catalog_manager.cc:6565] ReplaceTablet: tablet 0e8120281fe84e9a89f25d07c1d7b6a8 deleted and replaced by tablet 35148d28de80407c954839099623e817
I20260810 23:33:08.567905 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e8120281fe84e9a89f25d07c1d7b6a8 with delete_type TABLET_DATA_DELETED (replaced by tablet 35148d28de80407c954839099623e817) from {username='slave'} at 127.0.0.1:47546
I20260810 23:33:08.571903 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e8120281fe84e9a89f25d07c1d7b6a8 with delete_type TABLET_DATA_DELETED (replaced by tablet 35148d28de80407c954839099623e817) from {username='slave'} at 127.0.0.1:58598
W20260810 23:33:08.573172 12214 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 54c9310d98ad4d20b9a584bde8525227 (table default.Table_5e1f708104aa41118f94a7e35cecf800 [id=9e4c719f0c7a4f839aed37bc54648a36], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 54c9310d98ad4d20b9a584bde8525227 consensus state has no leader
I20260810 23:33:08.573290 12170 master-stress-test.cc:580] Killed test thread 9/10
I20260810 23:33:08.573604 12170 master-stress-test.cc:578] Killing test thread 10/10
I20260810 23:33:08.574047 12170 master-stress-test.cc:580] Killed test thread 10/10
I20260810 23:33:08.574467 12170 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu with pid 12658
I20260810 23:33:08.606693 15836 raft_consensus.cc:2726] T ca8c0d1dacc344cbacb0c5988f0e2221 P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:08.626396 13691 tablet_bootstrap.cc:492] T 00f68708a0b74a58b69581e8324a33f6 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:08.627342 13691 ts_tablet_manager.cc:1397] T 00f68708a0b74a58b69581e8324a33f6 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.283s user 0.012s sys 0.012s
I20260810 23:33:08.632994 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e8120281fe84e9a89f25d07c1d7b6a8 with delete_type TABLET_DATA_DELETED (replaced by tablet 35148d28de80407c954839099623e817) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:08.631330 13691 raft_consensus.cc:348] T 00f68708a0b74a58b69581e8324a33f6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.636312 13691 raft_consensus.cc:374] T 00f68708a0b74a58b69581e8324a33f6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:08.636806 13691 raft_consensus.cc:729] T 00f68708a0b74a58b69581e8324a33f6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:08.638304 13691 consensus_queue.cc:260] T 00f68708a0b74a58b69581e8324a33f6 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.644910 12229 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_cf212e8521974a72a6b86f2ea6ef2ffc [id=339a844c37ee460cb948d7e58264e5c8] alter complete (version 1)
I20260810 23:33:08.646857 13691 ts_tablet_manager.cc:1428] T 00f68708a0b74a58b69581e8324a33f6 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.019s user 0.003s sys 0.005s
I20260810 23:33:08.648551 13691 tablet_bootstrap.cc:492] T f1452c292e9c476b9e26f56998f0ad55 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:08.658134 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet 806fe5162aef4d36ad5e2ac324047426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:08.658114 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:04 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:08.660531 13665 tablet_bootstrap.cc:654] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:08.668411 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:04 UTC) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:08.670742 13025 tablet_service.cc:1514] Processing DeleteTablet for tablet a480093c481941f0bb35949c7606f37f with delete_type TABLET_DATA_DELETED (replaced by tablet 0703cf71490c4c71a95f55863583a03c) from {username='slave'} at 127.0.0.1:58598
I20260810 23:33:08.597450 15486 ts_tablet_manager.cc:1918] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:33:08.674299 15486 log.cc:1196] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/dce0ec83ed8943a0b0788505bb75d1df
I20260810 23:33:08.675125 15486 ts_tablet_manager.cc:1939] T dce0ec83ed8943a0b0788505bb75d1df P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:08.676604 13691 tablet_bootstrap.cc:654] T f1452c292e9c476b9e26f56998f0ad55 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:08.694854 13691 tablet_bootstrap.cc:492] T f1452c292e9c476b9e26f56998f0ad55 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:08.695377 13691 ts_tablet_manager.cc:1397] T f1452c292e9c476b9e26f56998f0ad55 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.002s
I20260810 23:33:08.698102 13691 raft_consensus.cc:348] T f1452c292e9c476b9e26f56998f0ad55 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.698855 13691 raft_consensus.cc:374] T f1452c292e9c476b9e26f56998f0ad55 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:08.699097 13691 raft_consensus.cc:729] T f1452c292e9c476b9e26f56998f0ad55 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:08.714982 13691 consensus_queue.cc:260] T f1452c292e9c476b9e26f56998f0ad55 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.718037 13691 ts_tablet_manager.cc:1428] T f1452c292e9c476b9e26f56998f0ad55 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
I20260810 23:33:08.719234 13691 tablet_bootstrap.cc:492] T 27c39d751e544d9a8afa26dc935a819a P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:08.722273 13406 heartbeater.cc:503] Master 127.11.226.189:42101 was elected leader, sending a full tablet report...
I20260810 23:33:08.727362 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:51 UTC) from {username='slave'} at 127.0.0.1:53280
W20260810 23:33:08.740322 12234 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 44) took 1188 ms (client timeout 1000 ms). Trace:
I20260810 23:33:08.746709 13665 tablet_bootstrap.cc:492] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:08.747298 13665 ts_tablet_manager.cc:1397] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.338s user 0.010s sys 0.010s
I20260810 23:33:08.750272 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet 32fad37a28e14903b8bde5aac6ee4e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:00 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:08.753321 13665 raft_consensus.cc:348] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:08.754252 13665 raft_consensus.cc:374] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:08.754523 13665 raft_consensus.cc:729] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:08.756404 13665 consensus_queue.cc:260] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:08.760632 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 346) took 2410 ms. Trace:
I20260810 23:33:08.761101 15486 rpcz_store.cc:276] 0810 23:33:06.350536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.350717 (+ 181us) service_pool.cc:227] Handling call
0810 23:33:08.275411 (+1924694us) ts_tablet_manager.cc:1145] Deleting tablet dce0ec83ed8943a0b0788505bb75d1df
0810 23:33:08.323188 (+ 47777us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.491991 (+168803us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.597096 (+105105us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.760542 (+163446us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21141,"spinlock_wait_cycles":52096,"tablet-delete.queue_time_us":1923867}
I20260810 23:33:08.732280 13691 tablet_bootstrap.cc:654] T 27c39d751e544d9a8afa26dc935a819a P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:08.762925 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 347) took 1950 ms. Trace:
I20260810 23:33:08.762908 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet dce0ec83ed8943a0b0788505bb75d1df (table default.Table_94e24b4b27d047a99df427c453fe1752 [id=e462d54230344223a457f708501d2d30]) successfully deleted
I20260810 23:33:08.750155 13141 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:08.767174 15486 rpcz_store.cc:276] 0810 23:33:06.812153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.813571 (+ 1418us) service_pool.cc:227] Handling call
0810 23:33:08.762315 (+1948744us) ts_tablet_manager.cc:1145] Deleting tablet 7293fddc208a4ee7ad85906f28f3be08
0810 23:33:08.762453 (+ 138us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.762844 (+ 391us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1948078}
W20260810 23:33:08.769151 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 7293fddc208a4ee7ad85906f28f3be08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7293fddc208a4ee7ad85906f28f3be08 already in progress: creating tablet
I20260810 23:33:08.743839 14121 ts_tablet_manager.cc:1918] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:33:08.770614 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 352) took 1690 ms. Trace:
I20260810 23:33:08.771106 14121 log.cc:1196] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/dce0ec83ed8943a0b0788505bb75d1df
I20260810 23:33:08.771044 15486 rpcz_store.cc:276] 0810 23:33:07.080003 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.080177 (+ 174us) service_pool.cc:227] Handling call
0810 23:33:08.770111 (+1689934us) ts_tablet_manager.cc:1145] Deleting tablet 46450c84a9054fbdbb5523ec835d43d2
0810 23:33:08.770165 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.770538 (+ 373us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1689185}
I20260810 23:33:08.772145 14121 ts_tablet_manager.cc:1939] T dce0ec83ed8943a0b0788505bb75d1df P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
W20260810 23:33:08.772850 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 46450c84a9054fbdbb5523ec835d43d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46450c84a9054fbdbb5523ec835d43d2 already in progress: creating tablet
I20260810 23:33:08.774197 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 358) took 1145 ms. Trace:
I20260810 23:33:08.781162 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 333) took 2404 ms. Trace:
I20260810 23:33:08.781900 14121 rpcz_store.cc:276] 0810 23:33:06.377084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.377271 (+ 187us) service_pool.cc:227] Handling call
0810 23:33:08.397466 (+2020195us) ts_tablet_manager.cc:1145] Deleting tablet dce0ec83ed8943a0b0788505bb75d1df
0810 23:33:08.397533 (+ 67us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.571283 (+173750us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.743413 (+172130us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.781065 (+ 37652us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2019332}
I20260810 23:33:08.786902 15836 raft_consensus.cc:482] T 00f68708a0b74a58b69581e8324a33f6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:08.787563 15836 raft_consensus.cc:504] T 00f68708a0b74a58b69581e8324a33f6 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.788411 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet dce0ec83ed8943a0b0788505bb75d1df (table default.Table_94e24b4b27d047a99df427c453fe1752 [id=e462d54230344223a457f708501d2d30]) successfully deleted
I20260810 23:33:08.790016 15836 leader_election.cc:288] T 00f68708a0b74a58b69581e8324a33f6 P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:08.787565 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 335) took 1997 ms. Trace:
I20260810 23:33:08.791849 14121 rpcz_store.cc:276] 0810 23:33:06.790582 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.790729 (+ 147us) service_pool.cc:227] Handling call
0810 23:33:08.787071 (+1996342us) ts_tablet_manager.cc:1145] Deleting tablet 7293fddc208a4ee7ad85906f28f3be08
0810 23:33:08.787125 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.787458 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1995679}
I20260810 23:33:08.792238 15486 rpcz_store.cc:276] 0810 23:33:07.629215 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.630520 (+ 1305us) service_pool.cc:227] Handling call
0810 23:33:08.773837 (+1143317us) ts_tablet_manager.cc:1145] Deleting tablet a41ddf9f6f71443ba75c97d2b010372c
0810 23:33:08.773887 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.774130 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1142558}
W20260810 23:33:08.793095 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 7293fddc208a4ee7ad85906f28f3be08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7293fddc208a4ee7ad85906f28f3be08 already in progress: creating tablet
W20260810 23:33:08.793838 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet a41ddf9f6f71443ba75c97d2b010372c because the tablet was not found. No further retry: Not found: Tablet not found: a41ddf9f6f71443ba75c97d2b010372c
I20260810 23:33:08.778082 13665 ts_tablet_manager.cc:1428] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.030s user 0.008s sys 0.001s
I20260810 23:33:08.795528 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 340) took 1797 ms. Trace:
I20260810 23:33:08.802217 13691 tablet_bootstrap.cc:492] T 27c39d751e544d9a8afa26dc935a819a P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:08.802781 13691 ts_tablet_manager.cc:1397] T 27c39d751e544d9a8afa26dc935a819a P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.005s
I20260810 23:33:08.795871 14121 rpcz_store.cc:276] 0810 23:33:06.998310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.998495 (+ 185us) service_pool.cc:227] Handling call
0810 23:33:08.795080 (+1796585us) ts_tablet_manager.cc:1145] Deleting tablet 46450c84a9054fbdbb5523ec835d43d2
0810 23:33:08.795131 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.795462 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1795944}
I20260810 23:33:08.807241 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 345) took 1048 ms. Trace:
W20260810 23:33:08.807524 12215 catalog_manager.cc:4821] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 46450c84a9054fbdbb5523ec835d43d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46450c84a9054fbdbb5523ec835d43d2 already in progress: creating tablet
I20260810 23:33:08.806247 13691 raft_consensus.cc:348] T 27c39d751e544d9a8afa26dc935a819a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.808877 13691 raft_consensus.cc:374] T 27c39d751e544d9a8afa26dc935a819a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:08.809229 13691 raft_consensus.cc:729] T 27c39d751e544d9a8afa26dc935a819a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:08.810143 13691 consensus_queue.cc:260] T 27c39d751e544d9a8afa26dc935a819a P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.811352 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f68708a0b74a58b69581e8324a33f6" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
W20260810 23:33:08.740804 12234 rpcz_store.cc:269] 0810 23:33:07.552263 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.690825 (+138562us) service_pool.cc:227] Handling call
0810 23:33:08.740213 (+1049388us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:08.267576 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:08.269177 (+ 1601us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:08.269239 (+ 62us) write_op.cc:435] Acquired schema lock
0810 23:33:08.269279 (+ 40us) tablet.cc:660] Decoding operations
0810 23:33:08.269880 (+ 601us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:08.270005 (+ 125us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:08.270093 (+ 88us) tablet.cc:683] Acquiring locks for 1 operations
0810 23:33:08.270465 (+ 372us) tablet.cc:699] Row locks acquired
0810 23:33:08.270515 (+ 50us) write_op.cc:260] PREPARE: finished
0810 23:33:08.270847 (+ 332us) write_op.cc:270] Start()
0810 23:33:08.271045 (+ 198us) write_op.cc:276] Timestamp: P: 1786404788270742 usec, L: 0
0810 23:33:08.271094 (+ 49us) op_driver.cc:331] REPLICATION: starting
0810 23:33:08.277285 (+ 6191us) log.cc:842] Serialized 536 byte log entry
0810 23:33:08.423649 (+146364us) op_driver.cc:444] REPLICATION: finished
0810 23:33:08.469619 (+ 45970us) write_op.cc:301] APPLY: starting
0810 23:33:08.469698 (+ 79us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:08.469824 (+ 126us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:08.469940 (+ 116us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:08.470198 (+ 258us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:08.470347 (+ 149us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0810 23:33:08.470390 (+ 43us) write_op.cc:312] APPLY: finished
0810 23:33:08.470625 (+ 235us) log.cc:842] Serialized 23 byte log entry
0810 23:33:08.470820 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:08.470936 (+ 116us) write_op.cc:454] Released schema lock
0810 23:33:08.470988 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:08.520825 (+ 49837us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:08.520897 (+ 72us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":45680,"apply.run_cpu_time_us":2336,"apply.run_wall_time_us":51601,"num_ops":1,"prepare.queue_time_us":14586,"prepare.run_cpu_time_us":4966,"prepare.run_wall_time_us":53750,"raft.queue_time_us":135,"raft.run_cpu_time_us":1385,"raft.run_wall_time_us":1386,"replication_time_us":152380,"thread_start_us":61475,"threads_started":2}]]}
I20260810 23:33:08.807708 14121 rpcz_store.cc:276] 0810 23:33:07.758510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.758776 (+ 266us) service_pool.cc:227] Handling call
0810 23:33:08.806872 (+1048096us) ts_tablet_manager.cc:1145] Deleting tablet 4cdec13ee70d402fa3a08119e7bf1313
0810 23:33:08.806921 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.807160 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046843}
W20260810 23:33:08.818329 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 4cdec13ee70d402fa3a08119e7bf1313 because the tablet was not found. No further retry: Not found: Tablet not found: 4cdec13ee70d402fa3a08119e7bf1313
I20260810 23:33:08.828626 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 361) took 1198 ms. Trace:
I20260810 23:33:08.829152 15486 rpcz_store.cc:276] 0810 23:33:07.630343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.633679 (+ 3336us) service_pool.cc:227] Handling call
0810 23:33:08.800945 (+1167266us) ts_tablet_manager.cc:1145] Deleting tablet 240adfa4be844518bbc9624d0e408789
0810 23:33:08.801003 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.828538 (+ 27535us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1166334}
I20260810 23:33:08.830786 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 346) took 1067 ms. Trace:
W20260810 23:33:08.830933 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 240adfa4be844518bbc9624d0e408789 because the tablet was not found. No further retry: Not found: Tablet not found: 240adfa4be844518bbc9624d0e408789
I20260810 23:33:08.831343 14121 rpcz_store.cc:276] 0810 23:33:07.763026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.769004 (+ 5978us) service_pool.cc:227] Handling call
0810 23:33:08.817584 (+1048580us) ts_tablet_manager.cc:1145] Deleting tablet 88c53785fdd24df79f138df6176e75d2
0810 23:33:08.817631 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.817842 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1047810}
I20260810 23:33:08.832686 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 348) took 1045 ms. Trace:
W20260810 23:33:08.833045 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 88c53785fdd24df79f138df6176e75d2 because the tablet was not found. No further retry: Not found: Tablet not found: 88c53785fdd24df79f138df6176e75d2
I20260810 23:33:08.833024 14121 rpcz_store.cc:276] 0810 23:33:07.787111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.787247 (+ 136us) service_pool.cc:227] Handling call
0810 23:33:08.832338 (+1045091us) ts_tablet_manager.cc:1145] Deleting tablet ab8d987109ba40198645881b449cb972
0810 23:33:08.832391 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.832618 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1044311}
W20260810 23:33:08.834462 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet ab8d987109ba40198645881b449cb972 because the tablet was not found. No further retry: Not found: Tablet not found: ab8d987109ba40198645881b449cb972
I20260810 23:33:08.835769 13691 ts_tablet_manager.cc:1428] T 27c39d751e544d9a8afa26dc935a819a P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.033s user 0.007s sys 0.001s
I20260810 23:33:08.838593 13691 tablet_bootstrap.cc:492] T afc0e5cb08c44f7b9662d63905d99b02 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:08.839993 13665 tablet_bootstrap.cc:492] T caee6f3203e049568c4c7cd25684c015 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:08.841240 14121 tablet_replica.cc:330] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:33:08.842654 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 359) took 1213 ms. Trace:
I20260810 23:33:08.843189 15486 rpcz_store.cc:276] 0810 23:33:07.629590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.631390 (+ 1800us) service_pool.cc:227] Handling call
0810 23:33:08.832390 (+1201000us) ts_tablet_manager.cc:1145] Deleting tablet f1e8311c941b43058384cee6d8263dc1
0810 23:33:08.832454 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.842568 (+ 10114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196330}
W20260810 23:33:08.845153 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet f1e8311c941b43058384cee6d8263dc1 because the tablet was not found. No further retry: Not found: Tablet not found: f1e8311c941b43058384cee6d8263dc1
I20260810 23:33:08.845336 14121 raft_consensus.cc:2229] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:08.847395 14121 raft_consensus.cc:2258] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:08.864208 14121 ts_tablet_manager.cc:1905] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:08.868244 15486 tablet_replica.cc:330] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:08.869519 15486 raft_consensus.cc:2229] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:08.872727 15486 raft_consensus.cc:2258] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:08.882678 13665 tablet_bootstrap.cc:654] T caee6f3203e049568c4c7cd25684c015 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:08.882813 13691 tablet_bootstrap.cc:654] T afc0e5cb08c44f7b9662d63905d99b02 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:08.893214 13691 tablet_bootstrap.cc:492] T afc0e5cb08c44f7b9662d63905d99b02 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:08.893723 13691 ts_tablet_manager.cc:1397] T afc0e5cb08c44f7b9662d63905d99b02 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.005s
I20260810 23:33:08.894677 15486 ts_tablet_manager.cc:1905] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:08.896421 13691 raft_consensus.cc:348] T afc0e5cb08c44f7b9662d63905d99b02 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.897174 13691 raft_consensus.cc:374] T afc0e5cb08c44f7b9662d63905d99b02 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:08.897445 13691 raft_consensus.cc:729] T afc0e5cb08c44f7b9662d63905d99b02 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:08.902887 13691 consensus_queue.cc:260] T afc0e5cb08c44f7b9662d63905d99b02 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.919140 13691 ts_tablet_manager.cc:1428] T afc0e5cb08c44f7b9662d63905d99b02 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260810 23:33:08.920323 13691 tablet_bootstrap.cc:492] T 422b25457d1549e186641fcedd8feab9 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:08.946367 13665 tablet_bootstrap.cc:492] T caee6f3203e049568c4c7cd25684c015 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:08.947081 14121 ts_tablet_manager.cc:1918] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:08.947613 14121 log.cc:1196] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/a41ddf9f6f71443ba75c97d2b010372c
I20260810 23:33:08.948043 13665 ts_tablet_manager.cc:1397] T caee6f3203e049568c4c7cd25684c015 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.108s user 0.015s sys 0.000s
I20260810 23:33:08.953840 13691 tablet_bootstrap.cc:654] T 422b25457d1549e186641fcedd8feab9 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:08.962574 13665 raft_consensus.cc:348] T caee6f3203e049568c4c7cd25684c015 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.965415 14121 ts_tablet_manager.cc:1939] T a41ddf9f6f71443ba75c97d2b010372c P f26517150f1c480a877f15aec71d020b: Deleting consensus metadata
I20260810 23:33:08.984411 13665 raft_consensus.cc:374] T caee6f3203e049568c4c7cd25684c015 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:08.985561 13665 raft_consensus.cc:729] T caee6f3203e049568c4c7cd25684c015 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:08.992736 13665 consensus_queue.cc:260] T caee6f3203e049568c4c7cd25684c015 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:08.993588 12229 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet 00f68708a0b74a58b69581e8324a33f6 (table default.Table_d134a55028664442b08b229593877973 [id=26be70827ca442a4b24a26015ffd952a]). Expected version 2 got 0
I20260810 23:33:08.994910 12229 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]). Expected version 1 got 0
I20260810 23:33:09.008011 13665 ts_tablet_manager.cc:1428] T caee6f3203e049568c4c7cd25684c015 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.059s user 0.007s sys 0.000s
I20260810 23:33:09.015825 13058 raft_consensus.cc:3037] T f1452c292e9c476b9e26f56998f0ad55 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:09.019690 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]). Expected version 1 got 0
I20260810 23:33:09.021453 15744 raft_consensus.cc:482] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:09.022055 15744 raft_consensus.cc:504] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:09.024762 15744 leader_election.cc:288] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:09.025697 13025 tablet_service.cc:1467] Processing CreateTablet for tablet 2cfe8fa607664ee8a5403eabe8f844b9 (DEFAULT_TABLE table=default.Table_b07cd9e36cc84081a4ac019dc2dcad7c [id=09ab2717b0c94e1caa955a381677e665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.026973 13025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cfe8fa607664ee8a5403eabe8f844b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.028501 13059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ff3bb1947c4f689353587396d878b8" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:33:09.031538 13212 leader_election.cc:341] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:09.040771 13665 tablet_bootstrap.cc:492] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:09.043170 13280 tablet_service.cc:1467] Processing CreateTablet for tablet 2cfe8fa607664ee8a5403eabe8f844b9 (DEFAULT_TABLE table=default.Table_b07cd9e36cc84081a4ac019dc2dcad7c [id=09ab2717b0c94e1caa955a381677e665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.051199 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 1432c9f43cd34983a5a072b9c800da87 (DEFAULT_TABLE table=default.Table_c28d62ee4c2a432b82a28cc0111c4e4d [id=af6bc38dfa954733a32b98d5543a3ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.052767 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1432c9f43cd34983a5a072b9c800da87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:08.978482 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 349) took 1188 ms. Trace:
I20260810 23:33:09.064575 14121 rpcz_store.cc:276] 0810 23:33:07.789864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.789986 (+ 122us) service_pool.cc:227] Handling call
0810 23:33:08.841001 (+1051015us) ts_tablet_manager.cc:1145] Deleting tablet a41ddf9f6f71443ba75c97d2b010372c
0810 23:33:08.841056 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.912878 (+ 71822us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.946783 (+ 33905us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:08.978386 (+ 31603us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050148}
I20260810 23:33:09.066469 12215 catalog_manager.cc:4841] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): tablet a41ddf9f6f71443ba75c97d2b010372c (table default.Table_73f2b1b207fb45f2adecbd37412c5e6f [id=c57f7c273a9841c1ac02523a733a383f]) successfully deleted
I20260810 23:33:09.067898 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 350) took 1276 ms. Trace:
I20260810 23:33:09.068591 14121 rpcz_store.cc:276] 0810 23:33:07.791813 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.791950 (+ 137us) service_pool.cc:227] Handling call
0810 23:33:09.067572 (+1275622us) ts_tablet_manager.cc:1145] Deleting tablet 4f3049bcdc6e43c795961c74f0d008f5
0810 23:33:09.067624 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:09.067830 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1272971}
W20260810 23:33:09.070554 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 4f3049bcdc6e43c795961c74f0d008f5 because the tablet was not found. No further retry: Not found: Tablet not found: 4f3049bcdc6e43c795961c74f0d008f5
I20260810 23:33:09.084765 12235 catalog_manager.cc:5495] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } }
I20260810 23:33:09.082350 13691 tablet_bootstrap.cc:492] T 422b25457d1549e186641fcedd8feab9 P f26517150f1c480a877f15aec71d020b: No bootstrap required, opened a new log
I20260810 23:33:09.089923 13691 ts_tablet_manager.cc:1397] T 422b25457d1549e186641fcedd8feab9 P f26517150f1c480a877f15aec71d020b: Time spent bootstrapping tablet: real 0.170s user 0.022s sys 0.034s
I20260810 23:33:09.091444 15744 raft_consensus.cc:482] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:09.092772 13691 raft_consensus.cc:348] T 422b25457d1549e186641fcedd8feab9 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:09.093554 13691 raft_consensus.cc:374] T 422b25457d1549e186641fcedd8feab9 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:09.093807 13691 raft_consensus.cc:729] T 422b25457d1549e186641fcedd8feab9 P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26517150f1c480a877f15aec71d020b, State: Initialized, Role: FOLLOWER
I20260810 23:33:09.094465 13691 consensus_queue.cc:260] T 422b25457d1549e186641fcedd8feab9 P f26517150f1c480a877f15aec71d020b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:09.071691 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 347) took 1286 ms. Trace:
I20260810 23:33:09.096365 14121 rpcz_store.cc:276] 0810 23:33:07.785552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.794708 (+ 9156us) service_pool.cc:227] Handling call
0810 23:33:09.071398 (+1276690us) ts_tablet_manager.cc:1145] Deleting tablet f1e8311c941b43058384cee6d8263dc1
0810 23:33:09.071447 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:09.071629 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1275972}
W20260810 23:33:09.098047 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet f1e8311c941b43058384cee6d8263dc1 because the tablet was not found. No further retry: Not found: Tablet not found: f1e8311c941b43058384cee6d8263dc1
I20260810 23:33:09.099390 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 351) took 1284 ms. Trace:
I20260810 23:33:09.099777 14121 rpcz_store.cc:276] 0810 23:33:07.815404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.815644 (+ 240us) service_pool.cc:227] Handling call
0810 23:33:09.098950 (+1283306us) ts_tablet_manager.cc:1145] Deleting tablet 4d0d9227d8574de8bd73041f3e8043a0
0810 23:33:09.099000 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:09.099304 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1282446}
I20260810 23:33:09.102025 15836 raft_consensus.cc:482] T 27c39d751e544d9a8afa26dc935a819a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:33:09.102878 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet 4d0d9227d8574de8bd73041f3e8043a0 because the tablet was not found. No further retry: Not found: Tablet not found: 4d0d9227d8574de8bd73041f3e8043a0
I20260810 23:33:09.102603 15836 raft_consensus.cc:504] T 27c39d751e544d9a8afa26dc935a819a P f26517150f1c480a877f15aec71d020b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:09.106057 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 1432c9f43cd34983a5a072b9c800da87 (DEFAULT_TABLE table=default.Table_c28d62ee4c2a432b82a28cc0111c4e4d [id=af6bc38dfa954733a32b98d5543a3ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.106613 13327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c39d751e544d9a8afa26dc935a819a" candidate_uuid: "f26517150f1c480a877f15aec71d020b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" is_pre_election: true
I20260810 23:33:09.107594 13327 raft_consensus.cc:2379] T 27c39d751e544d9a8afa26dc935a819a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f26517150f1c480a877f15aec71d020b in current term 1: Already voted for candidate f54aa78a60784a3bb2f5eb4bb2401ce3 in this term.
I20260810 23:33:09.109128 15836 leader_election.cc:288] T 27c39d751e544d9a8afa26dc935a819a P f26517150f1c480a877f15aec71d020b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807), 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059)
I20260810 23:33:09.110364 13017 tablet_service.cc:1467] Processing CreateTablet for tablet ecd72728a4d04a5888170aaa028394a9 (DEFAULT_TABLE table=default.Table_7b45b2e7e9c242039e2f372c0491fdf7 [id=f2d145b522f94058b2ca8c0491c13eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.111565 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd72728a4d04a5888170aaa028394a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.140718 13665 tablet_bootstrap.cc:654] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:09.143443 13282 tablet_service.cc:1467] Processing CreateTablet for tablet ecd72728a4d04a5888170aaa028394a9 (DEFAULT_TABLE table=default.Table_7b45b2e7e9c242039e2f372c0491fdf7 [id=f2d145b522f94058b2ca8c0491c13eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.092088 15744 raft_consensus.cc:504] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } } peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } }
I20260810 23:33:09.146015 13018 tablet_service.cc:1467] Processing CreateTablet for tablet fba7c1c98934456b9ee95dfde46ef0ef (DEFAULT_TABLE table=default.Table_5b5e98d980b14439a4826410a95f81cb [id=265353c26473442fad44e2b973d46680]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.147272 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba7c1c98934456b9ee95dfde46ef0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.148929 15744 leader_election.cc:288] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:09.149992 13058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62dc27b59ba469d99e8d11372aceff5" candidate_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26517150f1c480a877f15aec71d020b" is_pre_election: true
W20260810 23:33:09.151605 13212 leader_election.cc:341] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:09.157064 13691 ts_tablet_manager.cc:1428] T 422b25457d1549e186641fcedd8feab9 P f26517150f1c480a877f15aec71d020b: Time spent starting tablet: real 0.067s user 0.006s sys 0.000s
I20260810 23:33:09.158154 13691 tablet_bootstrap.cc:492] T bd921e48f22348639f7549d47b165185 P f26517150f1c480a877f15aec71d020b: Bootstrap starting.
I20260810 23:33:09.162065 14121 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58598 (request call id 352) took 1328 ms. Trace:
I20260810 23:33:09.162428 14121 rpcz_store.cc:276] 0810 23:33:07.833603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.833816 (+ 213us) service_pool.cc:227] Handling call
0810 23:33:09.160763 (+1326947us) ts_tablet_manager.cc:1145] Deleting tablet f935119e4d9b4accb79db98185c80aca
0810 23:33:09.161785 (+ 1022us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:09.162000 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1325981}
W20260810 23:33:09.163838 12215 catalog_manager.cc:4808] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079): delete failed for tablet f935119e4d9b4accb79db98185c80aca because the tablet was not found. No further retry: Not found: Tablet not found: f935119e4d9b4accb79db98185c80aca
I20260810 23:33:09.165598 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cfe8fa607664ee8a5403eabe8f844b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.165810 14121 tablet_replica.cc:330] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b: stopping tablet replica
I20260810 23:33:09.166996 14121 raft_consensus.cc:2229] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:09.167397 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1432c9f43cd34983a5a072b9c800da87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.087776 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 684caf5aaa8347e8b4e02323318fb0b7 (table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]). Expected version 1 got 0
I20260810 23:33:09.168417 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd72728a4d04a5888170aaa028394a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.169003 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet ba8d56858471421a9fbc361ee6aff28f (table default.Table_91daf907141c4d738a6e3d655692427c [id=9d36464c8de043e7a91673f011908f50]). Expected version 2 got 0
I20260810 23:33:09.170219 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]). Expected version 1 got 0
I20260810 23:33:09.172150 12235 catalog_manager.cc:5495] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } }
I20260810 23:33:09.178633 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 00f68708a0b74a58b69581e8324a33f6 (table default.Table_d134a55028664442b08b229593877973 [id=26be70827ca442a4b24a26015ffd952a]). Expected version 2 got 0
I20260810 23:33:09.184825 13023 tablet_service.cc:1467] Processing CreateTablet for tablet 403caf051714438bb77fc2fbefca15c5 (DEFAULT_TABLE table=default.Table_68bcc5970e834ef8bb810ac4f663c1d0 [id=956f3847e6384c07a3c61ee05e5ff9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.184890 13286 tablet_service.cc:1467] Processing CreateTablet for tablet fba7c1c98934456b9ee95dfde46ef0ef (DEFAULT_TABLE table=default.Table_5b5e98d980b14439a4826410a95f81cb [id=265353c26473442fad44e2b973d46680]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.186045 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403caf051714438bb77fc2fbefca15c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.186095 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba7c1c98934456b9ee95dfde46ef0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.167536 14121 raft_consensus.cc:2258] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:09.206636 14121 ts_tablet_manager.cc:1905] T 62705f99687340aaa1aca0942b28487f P f26517150f1c480a877f15aec71d020b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:09.213375 15744 consensus_queue.cc:1035] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:33:09.231272 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 403caf051714438bb77fc2fbefca15c5 (DEFAULT_TABLE table=default.Table_68bcc5970e834ef8bb810ac4f663c1d0 [id=956f3847e6384c07a3c61ee05e5ff9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.232652 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403caf051714438bb77fc2fbefca15c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.232584 13012 tablet_service.cc:1467] Processing CreateTablet for tablet 57bc55e5b70044cd84c878835fe225f0 (DEFAULT_TABLE table=default.Table_c608362e6ea74293b5e8a84421d98e10 [id=2be595c4bd574994afaa2e183e8c1a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.233865 13012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bc55e5b70044cd84c878835fe225f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:09.235322 15866 log.cc:924] Time spent T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: Append to log took a long time: real 0.177s user 0.031s sys 0.004s
I20260810 23:33:09.256950 12235 catalog_manager.cc:5495] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131). New cstate: current_term: 1 leader_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } health_report { overall_health: UNKNOWN } } }
I20260810 23:33:09.260353 12229 catalog_manager.cc:6040] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0: default.Table_4a7751c5b61c4332adeeb9ee00474b27 [id=2c8320255aab4036b5350d082f9e04ef] alter complete (version 2)
I20260810 23:33:09.278117 13691 tablet_bootstrap.cc:654] T bd921e48f22348639f7549d47b165185 P f26517150f1c480a877f15aec71d020b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:09.283521 13277 tablet_service.cc:1467] Processing CreateTablet for tablet 57bc55e5b70044cd84c878835fe225f0 (DEFAULT_TABLE table=default.Table_c608362e6ea74293b5e8a84421d98e10 [id=2be595c4bd574994afaa2e183e8c1a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.284778 13277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bc55e5b70044cd84c878835fe225f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.288890 13014 tablet_service.cc:1467] Processing CreateTablet for tablet 9740d8bfea2b4e18ba9559966a7f1e0f (DEFAULT_TABLE table=default.Table_4b640b4e6af94983b828197e5ece8d86 [id=32e343f820e2402d92a22104bf04b8ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.290156 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9740d8bfea2b4e18ba9559966a7f1e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.305352 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 9740d8bfea2b4e18ba9559966a7f1e0f (DEFAULT_TABLE table=default.Table_4b640b4e6af94983b828197e5ece8d86 [id=32e343f820e2402d92a22104bf04b8ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.306718 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9740d8bfea2b4e18ba9559966a7f1e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.315595 13274 tablet_service.cc:1467] Processing CreateTablet for tablet c9c84a370e7740649dab114582ebb4ab (DEFAULT_TABLE table=default.Table_9625301c8cda460981e80e753c785d4b [id=d79b564767e24f3b947e21b4e56e2e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.316880 13274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c84a370e7740649dab114582ebb4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.320895 13017 tablet_service.cc:1467] Processing CreateTablet for tablet c9c84a370e7740649dab114582ebb4ab (DEFAULT_TABLE table=default.Table_9625301c8cda460981e80e753c785d4b [id=d79b564767e24f3b947e21b4e56e2e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.322196 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c84a370e7740649dab114582ebb4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.336853 15680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e964330cf113403a8aa5b73ddd3b31d0)
I20260810 23:33:09.338196 15680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:33:09.319370 15663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e964330cf113403a8aa5b73ddd3b31d0. Latest consensus state: current_term: 10 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:33:09.351531 15663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:09.367589 15812 consensus_queue.cc:1035] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:33:09.369820 15680 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:33:09.371176 12495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 11 candidate_status { last_received { term: 10 index: 371 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:33:09.371302 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 11 candidate_status { last_received { term: 10 index: 371 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:33:09.376590 15210 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b; no voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:33:09.381282 15251 ts_manager.cc:194] Registered new tserver with Master: f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:09.383145 15486 ts_tablet_manager.cc:1918] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:09.372681 15867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:33:09.391359 15259 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" instance_seqno: 1786404736570809) as {username='slave'} at 127.11.226.129:49639; Asking this server to re-register.
W20260810 23:33:09.393047 15259 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:49639 (request call id 4) took 1717 ms (client timeout 1000 ms). Trace:
I20260810 23:33:09.389664 15680 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260810 23:33:09.397060 15867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:09.404650 15266 ts_descriptor.cc:144] Processing retry of TS registration from permanent_uuid: "f26517150f1c480a877f15aec71d020b" instance_seqno: 1786404739485999
I20260810 23:33:09.411537 15252 ts_descriptor.cc:144] Processing retry of TS registration from permanent_uuid: "f26517150f1c480a877f15aec71d020b" instance_seqno: 1786404739485999
I20260810 23:33:09.412448 15252 ts_manager.cc:194] Re-registered known tserver with Master: f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:33:09.415468 15252 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:53995 (request call id 7) took 1667 ms (client timeout 1000 ms). Trace:
I20260810 23:33:09.416348 15486 log.cc:1196] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/a480093c481941f0bb35949c7606f37f
I20260810 23:33:09.420634 12229 catalog_manager.cc:5587] TS f26517150f1c480a877f15aec71d020b (127.11.226.130:42079) does not have the latest schema for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]). Expected version 1 got 0
I20260810 23:33:09.422500 15486 ts_tablet_manager.cc:1939] T a480093c481941f0bb35949c7606f37f P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
W20260810 23:33:09.393520 15259 rpcz_store.cc:269] 0810 23:33:07.675284 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.675533 (+ 249us) service_pool.cc:227] Handling call
0810 23:33:09.386824 (+1711291us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54000104d4. stack: 00007fc34f5db033 00007fc34f5daf30 00007fc34f15c534 00000000004db2b4 00000000004db249 00007fc3564e7af0 00007fc35656af2a 00007fc358436d6c 00007fc358530c3e 00007fc356067fb4 00007fc356067f42 00007fc356067e71 00007fc356067dec 00007fc3560670a2 00007fc35431194c 00007fc354310d86
0810 23:33:09.392911 (+ 6087us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3819834752}
I20260810 23:33:09.409425 15266 ts_manager.cc:194] Re-registered known tserver with Master: f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:09.423605 15271 ts_descriptor.cc:144] Processing retry of TS registration from permanent_uuid: "f26517150f1c480a877f15aec71d020b" instance_seqno: 1786404739485999
I20260810 23:33:09.433281 15271 ts_manager.cc:194] Re-registered known tserver with Master: f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:33:09.455649 12217 connection.cc:535] client connection to 127.11.226.129:46059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:09.456219 12217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 2 similar messages]
I20260810 23:33:09.456445 12170 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu with pid 12897
W20260810 23:33:09.471854 15188 connection.cc:535] server connection from 127.11.226.129:49639 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:09.486703 12217 catalog_manager.cc:4584] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): CreateTablet RPC failed for tablet c9c84a370e7740649dab114582ebb4ab: Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111)
I20260810 23:33:09.492403 15270 ts_descriptor.cc:144] Processing retry of TS registration from permanent_uuid: "f26517150f1c480a877f15aec71d020b" instance_seqno: 1786404739485999
I20260810 23:33:09.493451 15270 ts_manager.cc:194] Re-registered known tserver with Master: f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:33:09.415856 15252 rpcz_store.cc:269] 0810 23:33:07.747873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.748283 (+ 410us) service_pool.cc:227] Handling call
0810 23:33:09.396329 (+1648046us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54000104d4. stack: 00007fc34f5db033 00007fc34f5daf30 00007fc34f15c534 00000000004db2b4 00000000004db249 00007fc3564e7af0 00007fc35656af2a 00007fc358436d6c 00007fc358530c3e 00007fc356067fb4 00007fc356067f42 00007fc356067e71 00007fc356067dec 00007fc3560670a2 00007fc35431194c 00007fc354310d86
0810 23:33:09.415349 (+ 19020us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3149320448}
I20260810 23:33:09.494428 15272 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" instance_seqno: 1786404736570809) as {username='slave'} at 127.11.226.129:49639; Asking this server to re-register.
W20260810 23:33:09.496893 15272 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:49639 (request call id 3) took 5213 ms (client timeout 1000 ms). Trace:
I20260810 23:33:09.497056 15260 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" instance_seqno: 1786404742829048) as {username='slave'} at 127.11.226.131:36249; Asking this server to re-register.
W20260810 23:33:09.497309 15272 rpcz_store.cc:269] 0810 23:33:04.282936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.283116 (+ 180us) service_pool.cc:227] Handling call
0810 23:33:09.426208 (+5143092us) spinlock_profiling.cc:243] Waited 5 s on lock 0x7b54000104d4. stack: 00007fc34f5db033 00007fc34f5daf30 00007fc34f15c534 00000000004db2b4 00000000004db249 00007fc3564e7af0 00007fc35656af2a 00007fc358436d6c 00007fc358530c3e 00007fc356067fb4 00007fc356067f42 00007fc356067e71 00007fc356067dec 00007fc3560670a2 00007fc35431194c 00007fc354310d86
0810 23:33:09.496253 (+ 70045us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11506443136}
W20260810 23:33:09.498456 15260 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:36249 (request call id 4) took 1782 ms (client timeout 1000 ms). Trace:
I20260810 23:33:09.499809 15269 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" instance_seqno: 1786404742829048) as {username='slave'} at 127.11.226.131:36249; Asking this server to re-register.
W20260810 23:33:09.500531 15269 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:36249 (request call id 3) took 3420 ms (client timeout 1000 ms). Trace:
W20260810 23:33:09.500849 15269 rpcz_store.cc:269] 0810 23:33:06.079925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.080267 (+ 342us) service_pool.cc:227] Handling call
0810 23:33:09.425030 (+3344763us) spinlock_profiling.cc:243] Waited 3.2 s on lock 0x7b54000104d4. stack: 00007fc34f5db033 00007fc34f5daf30 00007fc34f15c534 00000000004db2b4 00000000004db249 00007fc3564e7af0 00007fc35656af2a 00007fc358436d6c 00007fc358530c3e 00007fc356067fb4 00007fc356067f42 00007fc356067e71 00007fc356067dec 00007fc3560670a2 00007fc35431194c 00007fc354310d86
0810 23:33:09.500458 (+ 75428us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7356572032}
W20260810 23:33:09.499125 15188 connection.cc:441] server connection from 127.11.226.129:49639 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.11.226.129:49639 (request call id 3) could send its response
W20260810 23:33:09.503206 13213 connection.cc:535] client connection to 127.11.226.129:46059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:09.503846 13213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:09.498809 15260 rpcz_store.cc:269] 0810 23:33:07.716463 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.716713 (+ 250us) service_pool.cc:227] Handling call
0810 23:33:09.426679 (+1709966us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54000104d4. stack: 00007fc34f5db033 00007fc34f5daf30 00007fc34f15c534 00000000004db2b4 00000000004db249 00007fc3564e7af0 00007fc35656af2a 00007fc358436d6c 00007fc358530c3e 00007fc356067fb4 00007fc356067f42 00007fc356067e71 00007fc356067dec 00007fc3560670a2 00007fc35431194c 00007fc354310d86
0810 23:33:09.498355 (+ 71676us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2762153216}
W20260810 23:33:09.504907 15271 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:53995 (request call id 4) took 5175 ms (client timeout 1000 ms). Trace:
I20260810 23:33:09.495473 15275 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" instance_seqno: 1786404742829048) as {username='slave'} at 127.11.226.131:36249; Asking this server to re-register.
W20260810 23:33:09.508278 15275 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:36249 (request call id 2) took 5581 ms (client timeout 1000 ms). Trace:
W20260810 23:33:09.505824 15271 rpcz_store.cc:269] 0810 23:33:04.329989 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:04.330199 (+ 210us) service_pool.cc:227] Handling call
0810 23:33:09.420388 (+5090189us) spinlock_profiling.cc:243] Waited 4.93 s on lock 0x7b54000104d4. stack: 00007fc34f5db033 00007fc34f5daf30 00007fc34f15c534 00000000004db2b4 00000000004db249 00007fc3564e7af0 00007fc35656af2a 00007fc358436d6c 00007fc358530c3e 00007fc356067fb4 00007fc356067f42 00007fc356067e71 00007fc356067dec 00007fc3560670a2 00007fc35431194c 00007fc354310d86
0810 23:33:09.504394 (+ 84006us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11336675712}
I20260810 23:33:09.515411 15614 catalog_manager.cc:1259] Loaded cluster ID: f4f3e41035bf41dc9042550cc0b04e2a
I20260810 23:33:09.515738 15614 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: loading cluster ID for follower catalog manager: success
W20260810 23:33:09.509348 15275 rpcz_store.cc:269] 0810 23:33:03.927288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:03.927486 (+ 198us) service_pool.cc:227] Handling call
0810 23:33:09.424137 (+5496651us) spinlock_profiling.cc:243] Waited 5.07 s on lock 0x7b54000104d4. stack: 00007fc34f5db033 00007fc34f5daf30 00007fc34f15c534 00000000004db2b4 00000000004db249 00007fc3564e7af0 00007fc35656af2a 00007fc358436d6c 00007fc358530c3e 00007fc356067fb4 00007fc356067f42 00007fc356067e71 00007fc356067dec 00007fc3560670a2 00007fc35431194c 00007fc354310d86
0810 23:33:09.508201 (+ 84064us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11666343808}
W20260810 23:33:09.526242 15270 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:53995 (request call id 5) took 3981 ms (client timeout 1000 ms). Trace:
W20260810 23:33:09.519662 15266 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.130:53995 (request call id 6) took 2815 ms (client timeout 1000 ms). Trace:
I20260810 23:33:09.534407 13665 tablet_bootstrap.cc:492] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:09.534998 13665 ts_tablet_manager.cc:1397] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.494s user 0.008s sys 0.006s
I20260810 23:33:09.539526 13665 raft_consensus.cc:348] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:09.540408 13665 raft_consensus.cc:374] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:09.540691 13665 raft_consensus.cc:729] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
W20260810 23:33:09.526695 15270 rpcz_store.cc:269] 0810 23:33:05.546024 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:05.546419 (+ 395us) service_pool.cc:227] Handling call
0810 23:33:09.426290 (+3879871us) spinlock_profiling.cc:243] Waited 3.77 s on lock 0x7b54000104d4. stack: 00007fc34f5db033 00007fc34f5daf30 00007fc34f15c534 00000000004db2b4 00000000004db249 00007fc3564e7af0 00007fc35656af2a 00007fc358436d6c 00007fc358530c3e 00007fc356067fb4 00007fc356067f42 00007fc356067e71 00007fc356067dec 00007fc3560670a2 00007fc35431194c 00007fc354310d86
0810 23:33:09.526152 (+ 99862us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8660480256}
I20260810 23:33:09.544519 13665 consensus_queue.cc:260] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:33:09.540593 15266 rpcz_store.cc:269] 0810 23:33:06.703911 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:06.704722 (+ 811us) service_pool.cc:227] Handling call
0810 23:33:09.401103 (+2696381us) spinlock_profiling.cc:243] Waited 2.64 s on lock 0x7b54000104d4. stack: 00007fc34f5db033 00007fc34f5daf30 00007fc34f15c534 00000000004db2b4 00000000004db249 00007fc3564e7af0 00007fc35656af2a 00007fc358436d6c 00007fc358530c3e 00007fc356067fb4 00007fc356067f42 00007fc356067e71 00007fc356067dec 00007fc3560670a2 00007fc35431194c 00007fc354310d86
0810 23:33:09.519566 (+118463us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6075416832}
I20260810 23:33:09.549083 13665 ts_tablet_manager.cc:1428] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20260810 23:33:09.578071 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 363) took 1923 ms. Trace:
I20260810 23:33:09.578509 15486 rpcz_store.cc:276] 0810 23:33:07.654771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.654902 (+ 131us) service_pool.cc:227] Handling call
0810 23:33:08.867979 (+1213077us) ts_tablet_manager.cc:1145] Deleting tablet a480093c481941f0bb35949c7606f37f
0810 23:33:08.868084 (+ 105us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:08.970916 (+102832us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:09.381562 (+410646us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:09.577978 (+196416us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1212218}
I20260810 23:33:09.580796 13665 tablet_bootstrap.cc:492] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:09.584455 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet a480093c481941f0bb35949c7606f37f (table default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f]) successfully deleted
I20260810 23:33:09.586126 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 364) took 1885 ms. Trace:
I20260810 23:33:09.586632 15486 rpcz_store.cc:276] 0810 23:33:07.700718 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.700854 (+ 136us) service_pool.cc:227] Handling call
0810 23:33:09.585784 (+1884930us) ts_tablet_manager.cc:1145] Deleting tablet 62705f99687340aaa1aca0942b28487f
0810 23:33:09.585841 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:09.586055 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1884259}
W20260810 23:33:09.588279 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 62705f99687340aaa1aca0942b28487f because the tablet was not found. No further retry: Not found: Tablet not found: 62705f99687340aaa1aca0942b28487f
I20260810 23:33:09.589398 15486 tablet_replica.cc:330] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:09.590361 15486 raft_consensus.cc:2229] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:09.599798 13665 tablet_bootstrap.cc:654] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
W20260810 23:33:09.602502 13213 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 98750.3us
I20260810 23:33:09.605773 15486 raft_consensus.cc:2258] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:09.610013 15486 ts_tablet_manager.cc:1905] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:09.615497 15614 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: acquiring CA information for follower catalog manager: success
I20260810 23:33:09.621160 15300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37252 (request call id 0) took 6797 ms. Trace:
W20260810 23:33:09.626820 13213 connection.cc:535] server connection from 127.11.226.129:59455 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:33:09.632325 13665 tablet_bootstrap.cc:492] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:09.633038 13665 ts_tablet_manager.cc:1397] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.052s user 0.013s sys 0.000s
W20260810 23:33:09.651324 13213 consensus_peers.cc:487] T 4bce222cd5ab41fdb12fff3fe3dfbd3c P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:09.652247 13213 consensus_peers.cc:487] T f1452c292e9c476b9e26f56998f0ad55 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:09.659409 13277 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd with delete_type TABLET_DATA_DELETED (replaced by tablet 46450c84a9054fbdbb5523ec835d43d2) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:09.659840 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet a480093c481941f0bb35949c7606f37f with delete_type TABLET_DATA_DELETED (replaced by tablet 0703cf71490c4c71a95f55863583a03c) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:09.635944 13665 raft_consensus.cc:348] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:09.669711 13665 raft_consensus.cc:374] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:09.670017 13665 raft_consensus.cc:729] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:09.670806 13665 consensus_queue.cc:260] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:33:09.668351 13213 consensus_peers.cc:487] T 0d1b15dd545c4598aa5c7a9dc53618ec P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:09.674702 13213 consensus_peers.cc:487] T ff7bd9b6ca4845d8b8705ff970c8f06e P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:09.675168 13213 leader_election.cc:334] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111)
I20260810 23:33:09.675624 13213 leader_election.cc:302] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:09.676472 15744 raft_consensus.cc:2726] T 39ff3bb1947c4f689353587396d878b8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:33:09.677477 13213 consensus_peers.cc:487] T 4d1b9d20d09c4d1ba9a4b10959807e50 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:09.677913 13213 consensus_peers.cc:487] T 9443fb32d79941d58b13607e8ac064cd P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:09.678360 13213 consensus_peers.cc:487] T 0a0136a1e890436fae572b46e64750e7 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:09.678771 13213 leader_election.cc:334] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111)
I20260810 23:33:09.679208 13213 leader_election.cc:302] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f54aa78a60784a3bb2f5eb4bb2401ce3; no voters: 191d120f4c7f44fc8475305a6d2e75f3, f26517150f1c480a877f15aec71d020b
I20260810 23:33:09.685506 15259 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" instance_seqno: 1786404742829048) as {username='slave'} at 127.11.226.131:36249; Asking this server to re-register.
W20260810 23:33:09.659724 12235 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.11.226.131:49515 (request call id 45) took 874 ms (client timeout 1000 ms). Trace:
W20260810 23:33:09.688614 12235 rpcz_store.cc:269] 0810 23:33:08.784843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:08.785115 (+ 272us) service_pool.cc:227] Handling call
0810 23:33:09.330666 (+545551us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:09.330702 (+ 36us) write_op.cc:454] Released schema lock
0810 23:33:09.659639 (+328937us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:09.314441 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:09.314984 (+ 543us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:09.315017 (+ 33us) write_op.cc:435] Acquired schema lock
0810 23:33:09.315029 (+ 12us) tablet.cc:660] Decoding operations
0810 23:33:09.315728 (+ 699us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:09.315780 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:09.315846 (+ 66us) tablet.cc:683] Acquiring locks for 3 operations
0810 23:33:09.316202 (+ 356us) tablet.cc:699] Row locks acquired
0810 23:33:09.316218 (+ 16us) write_op.cc:260] PREPARE: finished
0810 23:33:09.316374 (+ 156us) write_op.cc:270] Start()
0810 23:33:09.316489 (+ 115us) write_op.cc:276] Timestamp: P: 1786404789316313 usec, L: 0
0810 23:33:09.316506 (+ 17us) op_driver.cc:331] REPLICATION: starting
0810 23:33:09.317213 (+ 707us) log.cc:842] Serialized 1302 byte log entry
0810 23:33:09.327778 (+ 10565us) op_driver.cc:444] REPLICATION: finished
0810 23:33:09.328203 (+ 425us) write_op.cc:301] APPLY: starting
0810 23:33:09.328288 (+ 85us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:09.328416 (+ 128us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:09.328516 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:09.328972 (+ 456us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:09.329258 (+ 286us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0810 23:33:09.329317 (+ 59us) write_op.cc:312] APPLY: finished
0810 23:33:09.329581 (+ 264us) log.cc:842] Serialized 35 byte log entry
0810 23:33:09.329780 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:09.329876 (+ 96us) write_op.cc:454] Released schema lock
0810 23:33:09.329916 (+ 40us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":243,"apply.run_cpu_time_us":1949,"apply.run_wall_time_us":1947,"num_ops":3,"prepare.queue_time_us":118,"prepare.run_cpu_time_us":3154,"prepare.run_wall_time_us":5572,"raft.queue_time_us":164,"raft.run_cpu_time_us":825,"raft.run_wall_time_us":824,"replication_time_us":11179}]]}
I20260810 23:33:09.692114 13406 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:09.702946 13665 ts_tablet_manager.cc:1428] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.069s user 0.007s sys 0.000s
W20260810 23:33:09.704933 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet ba8d56858471421a9fbc361ee6aff28f (table default.Table_91daf907141c4d738a6e3d655692427c [id=9d36464c8de043e7a91673f011908f50], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet ba8d56858471421a9fbc361ee6aff28f consensus state has no leader
I20260810 23:33:09.709653 15744 raft_consensus.cc:2726] T e62dc27b59ba469d99e8d11372aceff5 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:33:09.712862 13213 consensus_peers.cc:487] T 975a9bab969e46d2b939fc76ce350b99 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 23:33:09.729511 13213 consensus_peers.cc:487] T df3ac117f3b34c1bb55b6a258d4fd539 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:09.733733 13665 tablet_bootstrap.cc:492] T 31730595c84a44579c4e9db09ef603ba P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
W20260810 23:33:09.752924 13213 consensus_peers.cc:487] T 00f68708a0b74a58b69581e8324a33f6 P f54aa78a60784a3bb2f5eb4bb2401ce3 -> Peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Couldn't send request to peer 191d120f4c7f44fc8475305a6d2e75f3. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:09.763119 15259 ts_manager.cc:194] Registered new tserver with Master: f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807)
I20260810 23:33:09.766921 15486 ts_tablet_manager.cc:1918] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:09.767730 15486 log.cc:1196] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/806fe5162aef4d36ad5e2ac324047426
I20260810 23:33:09.768733 15486 ts_tablet_manager.cc:1939] T 806fe5162aef4d36ad5e2ac324047426 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:09.755798 13665 tablet_bootstrap.cc:654] T 31730595c84a44579c4e9db09ef603ba P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:09.785425 15614 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:33:09.792930 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]). Expected version 1 got 0
I20260810 23:33:09.793972 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]). Expected version 1 got 0
I20260810 23:33:09.794646 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet caee6f3203e049568c4c7cd25684c015 (table default.Table_767939c31e6e4afcaf3d29cef1e53985 [id=5fff4f293aaa42719e4cd0b3403b59ed]). Expected version 1 got 0
I20260810 23:33:09.801530 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet e62dc27b59ba469d99e8d11372aceff5 (table default.Table_3b251f8589be43f29704b8906d49f6cd [id=c56022cc8cad4c7cbefafad0782cefde]). Expected version 1 got 0
I20260810 23:33:09.802548 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 00f68708a0b74a58b69581e8324a33f6 (table default.Table_d134a55028664442b08b229593877973 [id=26be70827ca442a4b24a26015ffd952a]). Expected version 2 got 0
I20260810 23:33:09.812392 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 365) took 2043 ms. Trace:
I20260810 23:33:09.812846 15486 rpcz_store.cc:276] 0810 23:33:07.769252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.769388 (+ 136us) service_pool.cc:227] Handling call
0810 23:33:09.589216 (+1819828us) ts_tablet_manager.cc:1145] Deleting tablet 806fe5162aef4d36ad5e2ac324047426
0810 23:33:09.589264 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:09.695831 (+106567us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:09.735868 (+ 40037us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:09.812300 (+ 76432us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1819151}
I20260810 23:33:09.814242 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 357) took 2185 ms. Trace:
I20260810 23:33:09.814589 15486 rpcz_store.cc:276] 0810 23:33:07.628807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:07.781321 (+152514us) service_pool.cc:227] Handling call
0810 23:33:09.813908 (+2032587us) ts_tablet_manager.cc:1145] Deleting tablet ab8d987109ba40198645881b449cb972
0810 23:33:09.813961 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:09.814180 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1940077}
I20260810 23:33:09.815235 13274 tablet_service.cc:1467] Processing CreateTablet for tablet 35148d28de80407c954839099623e817 (DEFAULT_TABLE table=default.Table_a7d468b7f476404c8b400f3923cd1543 [id=bc138fc988a942b093d18a66e0cf605c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:09.816040 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b54edbb45d496c94efedb6feee8e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:45 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:09.816447 13274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35148d28de80407c954839099623e817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:09.815829 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 367) took 1250 ms. Trace:
I20260810 23:33:09.816715 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 806fe5162aef4d36ad5e2ac324047426 (table default.Table_848da5bb390440608b71e81db171df2c [id=6920efe096de4514a4d75aa2871876b9]) successfully deleted
I20260810 23:33:09.816838 15486 rpcz_store.cc:276] 0810 23:33:08.565745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:08.565923 (+ 178us) service_pool.cc:227] Handling call
0810 23:33:09.815462 (+1249539us) ts_tablet_manager.cc:1145] Deleting tablet 0e8120281fe84e9a89f25d07c1d7b6a8
0810 23:33:09.815506 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:09.815771 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1182002}
W20260810 23:33:09.817430 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet ab8d987109ba40198645881b449cb972 because the tablet was not found. No further retry: Not found: Tablet not found: ab8d987109ba40198645881b449cb972
I20260810 23:33:09.818137 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 368) took 1162 ms. Trace:
I20260810 23:33:09.818447 15486 rpcz_store.cc:276] 0810 23:33:08.655483 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:08.656287 (+ 804us) service_pool.cc:227] Handling call
0810 23:33:09.817866 (+1161579us) ts_tablet_manager.cc:1145] Deleting tablet 8fb2ee1c9b41478da001aa67d7a7ad3c
0810 23:33:09.817916 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:09.818088 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1159172}
I20260810 23:33:09.819520 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 369) took 1102 ms. Trace:
I20260810 23:33:09.819823 15486 rpcz_store.cc:276] 0810 23:33:08.717169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:08.717318 (+ 149us) service_pool.cc:227] Handling call
0810 23:33:09.819272 (+1101954us) ts_tablet_manager.cc:1145] Deleting tablet 1b1f24e226b94ccf8ad5a656c80a5b3b
0810 23:33:09.819317 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:09.819468 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1091332}
I20260810 23:33:09.821362 15744 raft_consensus.cc:482] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:09.821867 15744 raft_consensus.cc:504] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:09.815826 13277 tablet_service.cc:1514] Processing DeleteTablet for tablet 422b25457d1549e186641fcedd8feab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:00 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:09.828132 15486 tablet_replica.cc:330] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:09.829905 15486 raft_consensus.cc:2229] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:09.830538 15486 raft_consensus.cc:2258] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260810 23:33:09.831707 12214 catalog_manager.cc:4821] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 0e8120281fe84e9a89f25d07c1d7b6a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e8120281fe84e9a89f25d07c1d7b6a8 already in progress: creating tablet
W20260810 23:33:09.832580 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 8fb2ee1c9b41478da001aa67d7a7ad3c because the tablet was not found. No further retry: Not found: Tablet not found: 8fb2ee1c9b41478da001aa67d7a7ad3c
W20260810 23:33:09.833222 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet 1b1f24e226b94ccf8ad5a656c80a5b3b because the tablet was not found. No further retry: Not found: Tablet not found: 1b1f24e226b94ccf8ad5a656c80a5b3b
I20260810 23:33:09.834761 13665 tablet_bootstrap.cc:492] T 31730595c84a44579c4e9db09ef603ba P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:09.835330 15744 leader_election.cc:288] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:09.839641 13665 ts_tablet_manager.cc:1397] T 31730595c84a44579c4e9db09ef603ba P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.106s user 0.016s sys 0.000s
W20260810 23:33:09.839867 13213 leader_election.cc:334] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111)
I20260810 23:33:09.840194 15486 ts_tablet_manager.cc:1905] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:09.842288 13665 raft_consensus.cc:348] T 31730595c84a44579c4e9db09ef603ba P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:09.843029 13665 raft_consensus.cc:374] T 31730595c84a44579c4e9db09ef603ba P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:09.843281 13665 raft_consensus.cc:729] T 31730595c84a44579c4e9db09ef603ba P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:09.843904 13665 consensus_queue.cc:260] T 31730595c84a44579c4e9db09ef603ba P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:09.846560 13665 ts_tablet_manager.cc:1428] T 31730595c84a44579c4e9db09ef603ba P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 23:33:09.847642 13665 tablet_bootstrap.cc:492] T a6f53aca31ee4e338d9b5e281557c121 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
W20260810 23:33:09.855947 12217 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.226.129:46059, user_credentials={real_user=slave}} blocked reactor thread for 46580.5us
I20260810 23:33:09.857069 13665 tablet_bootstrap.cc:654] T a6f53aca31ee4e338d9b5e281557c121 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:09.865656 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]). Expected version 1 got 0
I20260810 23:33:09.898278 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet f1452c292e9c476b9e26f56998f0ad55 (table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]). Expected version 1 got 0
I20260810 23:33:09.901775 13665 tablet_bootstrap.cc:492] T a6f53aca31ee4e338d9b5e281557c121 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:09.902439 13665 ts_tablet_manager.cc:1397] T a6f53aca31ee4e338d9b5e281557c121 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.000s
I20260810 23:33:09.905551 13665 raft_consensus.cc:348] T a6f53aca31ee4e338d9b5e281557c121 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:09.907344 13665 raft_consensus.cc:374] T a6f53aca31ee4e338d9b5e281557c121 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:09.909642 13665 raft_consensus.cc:729] T a6f53aca31ee4e338d9b5e281557c121 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:09.910633 13665 consensus_queue.cc:260] T a6f53aca31ee4e338d9b5e281557c121 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:09.924698 15764 raft_consensus.cc:482] T caee6f3203e049568c4c7cd25684c015 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:09.925304 15764 raft_consensus.cc:504] T caee6f3203e049568c4c7cd25684c015 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:09.930727 15764 leader_election.cc:288] T caee6f3203e049568c4c7cd25684c015 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:33:09.934271 13213 leader_election.cc:334] T caee6f3203e049568c4c7cd25684c015 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111)
I20260810 23:33:09.938584 13665 ts_tablet_manager.cc:1428] T a6f53aca31ee4e338d9b5e281557c121 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20260810 23:33:09.919876 15486 ts_tablet_manager.cc:1918] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:09.941059 15486 log.cc:1196] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/32fad37a28e14903b8bde5aac6ee4e69
I20260810 23:33:09.942050 15486 ts_tablet_manager.cc:1939] T 32fad37a28e14903b8bde5aac6ee4e69 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:09.955202 13665 tablet_bootstrap.cc:492] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:09.963826 15486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53280 (request call id 370) took 1218 ms. Trace:
I20260810 23:33:09.964836 15486 rpcz_store.cc:276] 0810 23:33:08.745325 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:08.745514 (+ 189us) service_pool.cc:227] Handling call
0810 23:33:09.820690 (+1075176us) ts_tablet_manager.cc:1145] Deleting tablet 32fad37a28e14903b8bde5aac6ee4e69
0810 23:33:09.821917 (+ 1227us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0810 23:33:09.879907 (+ 57990us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:09.919527 (+ 39620us) tablet_metadata.cc:641] Metadata flushed
0810 23:33:09.963721 (+ 44194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1069873}
I20260810 23:33:09.966533 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 32fad37a28e14903b8bde5aac6ee4e69 (table default.Table_2f4392eef1c74daa870d52081ffb3c90 [id=9290d4418f5a4ca9a7d7768dae5843a9]) successfully deleted
I20260810 23:33:09.974103 15486 tablet_replica.cc:330] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:09.974753 13665 tablet_bootstrap.cc:654] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:09.975330 15486 raft_consensus.cc:2229] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:09.975989 15486 raft_consensus.cc:2258] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:09.978327 15486 ts_tablet_manager.cc:1905] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:09.985476 13665 tablet_bootstrap.cc:492] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:09.986090 13665 ts_tablet_manager.cc:1397] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.000s
I20260810 23:33:09.990988 15897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e964330cf113403a8aa5b73ddd3b31d0)
I20260810 23:33:09.991566 15897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:33:09.993904 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet a6f53aca31ee4e338d9b5e281557c121 (table default.Table_7e1011c9a03343acb3bdf04b07eb962e [id=89c842ab64c546bbb967a2246a7220ac]). Expected version 3 got 0
I20260810 23:33:10.006419 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 11 candidate_status { last_received { term: 10 index: 371 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:33:10.014353 13665 raft_consensus.cc:348] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:10.015156 13665 raft_consensus.cc:374] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:10.014588 12495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198f534261994047a68e1d509bb2fd8b" candidate_term: 11 candidate_status { last_received { term: 10 index: 371 } } ignore_live_leader: false dest_uuid: "681674cb24274f10b200f264f155c2d1" is_pre_election: true
I20260810 23:33:10.015417 13665 raft_consensus.cc:729] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:10.015950 15897 leader_election.cc:288] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:33:10.016222 13665 consensus_queue.cc:260] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:10.017093 15210 leader_election.cc:302] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b; no voters: 681674cb24274f10b200f264f155c2d1, e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:33:10.020404 15897 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260810 23:33:10.019950 13665 ts_tablet_manager.cc:1428] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20260810 23:33:10.028981 13665 tablet_bootstrap.cc:492] T ad8be93f60644e16b28453b2fd6c0285 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:10.031068 15764 raft_consensus.cc:482] T 31730595c84a44579c4e9db09ef603ba P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:10.031898 15764 raft_consensus.cc:504] T 31730595c84a44579c4e9db09ef603ba P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
W20260810 23:33:10.039309 13213 leader_election.cc:334] T 31730595c84a44579c4e9db09ef603ba P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111)
I20260810 23:33:10.041966 15486 ts_tablet_manager.cc:1918] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:10.042488 15764 leader_election.cc:288] T 31730595c84a44579c4e9db09ef603ba P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:10.046018 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet a6f53aca31ee4e338d9b5e281557c121 (table default.Table_7e1011c9a03343acb3bdf04b07eb962e [id=89c842ab64c546bbb967a2246a7220ac]). Expected version 3 got 0
I20260810 23:33:10.042526 15486 log.cc:1196] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/2c6a6e6fb6124c4bbdae70eb444f6fdd
I20260810 23:33:10.050467 15486 ts_tablet_manager.cc:1939] T 2c6a6e6fb6124c4bbdae70eb444f6fdd P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:10.071837 13665 tablet_bootstrap.cc:654] T ad8be93f60644e16b28453b2fd6c0285 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:10.072672 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 2c6a6e6fb6124c4bbdae70eb444f6fdd (table default.Table_9625301c8cda460981e80e753c785d4b [id=d79b564767e24f3b947e21b4e56e2e3b]) successfully deleted
W20260810 23:33:10.078048 12214 catalog_manager.cc:4808] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): delete failed for tablet a480093c481941f0bb35949c7606f37f because the tablet was not found. No further retry: Not found: Tablet not found: a480093c481941f0bb35949c7606f37f
I20260810 23:33:10.078614 15486 tablet_replica.cc:330] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:10.079594 15486 raft_consensus.cc:2229] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:10.080219 15486 raft_consensus.cc:2258] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:10.082561 15486 ts_tablet_manager.cc:1905] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:10.103276 13274 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be35dabbec44caebd1df687b167f34a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='slave'} at 127.0.0.1:53280
I20260810 23:33:10.112067 13665 tablet_bootstrap.cc:492] T ad8be93f60644e16b28453b2fd6c0285 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:10.112653 13665 ts_tablet_manager.cc:1397] T ad8be93f60644e16b28453b2fd6c0285 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.084s user 0.015s sys 0.002s
I20260810 23:33:10.118782 15764 raft_consensus.cc:482] T a6f53aca31ee4e338d9b5e281557c121 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:10.119338 15764 raft_consensus.cc:504] T a6f53aca31ee4e338d9b5e281557c121 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:10.121600 15764 leader_election.cc:288] T a6f53aca31ee4e338d9b5e281557c121 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:33:10.126919 13213 leader_election.cc:334] T a6f53aca31ee4e338d9b5e281557c121 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111)
I20260810 23:33:10.127331 15764 raft_consensus.cc:482] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:10.127905 15764 raft_consensus.cc:504] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:10.130048 15764 leader_election.cc:288] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
I20260810 23:33:10.115392 13665 raft_consensus.cc:348] T ad8be93f60644e16b28453b2fd6c0285 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:10.131793 13665 raft_consensus.cc:374] T ad8be93f60644e16b28453b2fd6c0285 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:10.132119 13665 raft_consensus.cc:729] T ad8be93f60644e16b28453b2fd6c0285 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
W20260810 23:33:10.134166 13213 leader_election.cc:334] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111)
I20260810 23:33:10.134879 15486 ts_tablet_manager.cc:1918] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:10.135418 15486 log.cc:1196] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/80b54edbb45d496c94efedb6feee8e45
I20260810 23:33:10.136425 15486 ts_tablet_manager.cc:1939] T 80b54edbb45d496c94efedb6feee8e45 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:10.146008 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 80b54edbb45d496c94efedb6feee8e45 (table default.Table_cc87f3e94dc84a44822a6f3af8f0c1c3 [id=2141df5169c64180b39b058b342f779d]) successfully deleted
I20260810 23:33:10.147142 13665 consensus_queue.cc:260] T ad8be93f60644e16b28453b2fd6c0285 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:10.147198 15486 tablet_replica.cc:330] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:10.152192 13665 ts_tablet_manager.cc:1428] T ad8be93f60644e16b28453b2fd6c0285 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.039s user 0.008s sys 0.000s
I20260810 23:33:10.166113 13665 tablet_bootstrap.cc:492] T d5573b3e7e5b47f9885fc1807202d2e6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:10.173698 15486 raft_consensus.cc:2229] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:10.174387 15486 raft_consensus.cc:2258] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:10.176648 15486 ts_tablet_manager.cc:1905] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:10.188931 13665 tablet_bootstrap.cc:654] T d5573b3e7e5b47f9885fc1807202d2e6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:10.245134 13665 tablet_bootstrap.cc:492] T d5573b3e7e5b47f9885fc1807202d2e6 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:10.245819 13665 ts_tablet_manager.cc:1397] T d5573b3e7e5b47f9885fc1807202d2e6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.080s user 0.013s sys 0.000s
I20260810 23:33:10.268221 13665 raft_consensus.cc:348] T d5573b3e7e5b47f9885fc1807202d2e6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:10.272217 13665 raft_consensus.cc:374] T d5573b3e7e5b47f9885fc1807202d2e6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:10.280164 13665 raft_consensus.cc:729] T d5573b3e7e5b47f9885fc1807202d2e6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:10.284335 13665 consensus_queue.cc:260] T d5573b3e7e5b47f9885fc1807202d2e6 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:10.290836 13665 ts_tablet_manager.cc:1428] T d5573b3e7e5b47f9885fc1807202d2e6 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.045s user 0.006s sys 0.000s
I20260810 23:33:10.303433 15486 ts_tablet_manager.cc:1918] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:10.303958 15486 log.cc:1196] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/422b25457d1549e186641fcedd8feab9
I20260810 23:33:10.304772 15486 ts_tablet_manager.cc:1939] T 422b25457d1549e186641fcedd8feab9 P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:10.330713 15901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e964330cf113403a8aa5b73ddd3b31d0)
I20260810 23:33:10.335309 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 4d1b9d20d09c4d1ba9a4b10959807e50 (table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]). Expected version 1 got 0
I20260810 23:33:10.341284 13665 tablet_bootstrap.cc:492] T 7276caee8a674131860ef17898c65308 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:10.343508 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 422b25457d1549e186641fcedd8feab9 (table default.Table_3f5d5fb567214af3bcd7557741d52620 [id=51cbefc808e244819a59d344cb37c02e]) successfully deleted
I20260810 23:33:10.344581 15486 tablet_replica.cc:330] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3: stopping tablet replica
I20260810 23:33:10.345614 15486 raft_consensus.cc:2229] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:10.346220 15486 raft_consensus.cc:2258] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:10.348886 15486 ts_tablet_manager.cc:1905] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:10.359911 13665 tablet_bootstrap.cc:654] T 7276caee8a674131860ef17898c65308 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:10.331485 15901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:33:10.375809 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 447 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0" is_pre_election: true
I20260810 23:33:10.378640 15901 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:33:10.379921 15296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 447 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:33:10.380702 15296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 10.
I20260810 23:33:10.382464 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1; no voters: e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:33:10.383409 15901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260810 23:33:10.383895 15901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Starting leader election (detected failure of leader e964330cf113403a8aa5b73ddd3b31d0)
I20260810 23:33:10.385326 15901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 10 FOLLOWER]: Advancing to term 11
I20260810 23:33:10.412274 13665 tablet_bootstrap.cc:492] T 7276caee8a674131860ef17898c65308 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:10.413082 13665 ts_tablet_manager.cc:1397] T 7276caee8a674131860ef17898c65308 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.072s user 0.013s sys 0.001s
I20260810 23:33:10.415946 13665 raft_consensus.cc:348] T 7276caee8a674131860ef17898c65308 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:10.430152 13665 raft_consensus.cc:374] T 7276caee8a674131860ef17898c65308 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:10.436793 13665 raft_consensus.cc:729] T 7276caee8a674131860ef17898c65308 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54aa78a60784a3bb2f5eb4bb2401ce3, State: Initialized, Role: FOLLOWER
I20260810 23:33:10.437634 13665 consensus_queue.cc:260] T 7276caee8a674131860ef17898c65308 P f54aa78a60784a3bb2f5eb4bb2401ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:10.449733 13665 ts_tablet_manager.cc:1428] T 7276caee8a674131860ef17898c65308 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20260810 23:33:10.469575 13665 tablet_bootstrap.cc:492] T f0cba350e63e46a894a89ccc5727aab8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Bootstrap starting.
I20260810 23:33:10.478031 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 7276caee8a674131860ef17898c65308 (table default.Table_adde4ae3b1394291be75cc80db4f457d [id=a02006db831d417597d63c0e7c9cad8c]). Expected version 1 got 0
I20260810 23:33:10.485773 13665 tablet_bootstrap.cc:654] T f0cba350e63e46a894a89ccc5727aab8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:10.488991 15486 ts_tablet_manager.cc:1918] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:10.489424 15486 log.cc:1196] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/0be35dabbec44caebd1df687b167f34a
I20260810 23:33:10.490160 15486 ts_tablet_manager.cc:1939] T 0be35dabbec44caebd1df687b167f34a P f54aa78a60784a3bb2f5eb4bb2401ce3: Deleting consensus metadata
I20260810 23:33:10.492404 12214 catalog_manager.cc:4841] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807): tablet 0be35dabbec44caebd1df687b167f34a (table default.Table_216b21581f454466a84891c7e3d0e567 [id=ff7ae9781f734fe1bd9565c344b6d5e4]) successfully deleted
W20260810 23:33:10.506592 12217 catalog_manager.cc:4584] TS 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): CreateTablet RPC failed for tablet fba7c1c98934456b9ee95dfde46ef0ef: Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111)
I20260810 23:33:10.510511 15901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:33:10.520265 12495 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 11 FOLLOWER]: Rejecting Update request from peer e964330cf113403a8aa5b73ddd3b31d0 for earlier term 10. Current term is 11. Ops: []
I20260810 23:33:10.523443 15903 consensus_queue.cc:1046] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 }, Status: INVALID_TERM, Last received: 10.447, Next index: 448, Last known committed idx: 447, Time since last communication: 0.001s
I20260810 23:33:10.524679 15744 raft_consensus.cc:482] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:10.525249 15744 raft_consensus.cc:504] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:10.527474 15744 leader_election.cc:288] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059), f26517150f1c480a877f15aec71d020b (127.11.226.130:42079)
W20260810 23:33:10.528060 15183 connection.cc:535] server connection from 127.11.226.130:53995 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:10.536280 13213 leader_election.cc:334] T afc0e5cb08c44f7b9662d63905d99b02 P f54aa78a60784a3bb2f5eb4bb2401ce3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 191d120f4c7f44fc8475305a6d2e75f3 (127.11.226.129:46059): Network error: Client connection negotiation failed: client connection to 127.11.226.129:46059: connect: Connection refused (error 111)
I20260810 23:33:10.545755 13665 tablet_bootstrap.cc:492] T f0cba350e63e46a894a89ccc5727aab8 P f54aa78a60784a3bb2f5eb4bb2401ce3: No bootstrap required, opened a new log
I20260810 23:33:10.546327 13665 ts_tablet_manager.cc:1397] T f0cba350e63e46a894a89ccc5727aab8 P f54aa78a60784a3bb2f5eb4bb2401ce3: Time spent bootstrapping tablet: real 0.077s user 0.010s sys 0.006s
I20260810 23:33:10.546175 12235 catalog_manager.cc:5587] TS f54aa78a60784a3bb2f5eb4bb2401ce3 (127.11.226.131:33807) does not have the latest schema for tablet 7276caee8a674131860ef17898c65308 (table default.Table_adde4ae3b1394291be75cc80db4f457d [id=a02006db831d417597d63c0e7c9cad8c]). Expected version 1 got 0
W20260810 23:33:10.548364 13212 connection.cc:535] client connection to 127.11.226.130:42079 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:10.548395 12215 connection.cc:535] client connection to 127.11.226.130:42079 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:10.548529 12218 connection.cc:535] server connection from 127.11.226.130:47217 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:33:10.555724 12254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 447 } } ignore_live_leader: false dest_uuid: "e964330cf113403a8aa5b73ddd3b31d0"
I20260810 23:33:10.561427 13665 raft_consensus.cc:348] T f0cba350e63e46a894a89ccc5727aab8 P f54aa78a60784a3bb2f5eb4bb2401ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54aa78a60784a3bb2f5eb4bb2401ce3" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 33807 } } peers { permanent_uuid: "191d120f4c7f44fc8475305a6d2e75f3" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 46059 } } peers { permanent_uuid: "f26517150f1c480a877f15aec71d020b" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42079 } }
I20260810 23:33:10.562024 12170 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu with pid 13149
I20260810 23:33:10.590723 15296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "681674cb24274f10b200f264f155c2d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 447 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b"
I20260810 23:33:10.591697 15296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 10 FOLLOWER]: Advancing to term 11
I20260810 23:33:10.604990 15901 leader_election.cc:288] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 election: Requested vote from peers e964330cf113403a8aa5b73ddd3b31d0 (127.11.226.190:43533), 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101)
I20260810 23:33:10.606896 15296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681674cb24274f10b200f264f155c2d1 in term 11.
I20260810 23:33:10.608389 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, 681674cb24274f10b200f264f155c2d1; no voters: e964330cf113403a8aa5b73ddd3b31d0
I20260810 23:33:10.614475 15905 raft_consensus.cc:3032] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 10 LEADER]: Stepping down as leader of term 10
I20260810 23:33:10.615058 15905 raft_consensus.cc:729] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e964330cf113403a8aa5b73ddd3b31d0, State: Running, Role: LEADER
I20260810 23:33:10.624861 15905 consensus_queue.cc:260] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 447, Committed index: 447, Last appended: 10.447, Last appended by leader: 447, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:33:10.626621 15905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 10 FOLLOWER]: Advancing to term 11
I20260810 23:33:10.627017 15901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 11 FOLLOWER]: Leader election won for term 11
I20260810 23:33:10.640180 15901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 11 LEADER]: Becoming Leader. State: Replica: 681674cb24274f10b200f264f155c2d1, State: Running, Role: LEADER
I20260810 23:33:10.641613 15901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 447, Committed index: 447, Last appended: 10.447, Last appended by leader: 447, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:33:10.658990 15908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 681674cb24274f10b200f264f155c2d1. Latest consensus state: current_term: 11 leader_uuid: "681674cb24274f10b200f264f155c2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:33:10.668866 15908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:10.684520 15909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:33:10.807725 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_35fbbca76d6c43419846a96203e57279 [id=01a719299d5b4f518bfd8d59a7ab211f]
I20260810 23:33:10.810112 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_d664d07d116e4f2aa4fc263449eb8dd0 [id=04ed6428e4df49b2ab0438035da70731]
I20260810 23:33:10.830605 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_c4f98a2612f6454fb7ca19de9ba29ab3 [id=092fca6f55f1456cb053678e1bb500cd]
I20260810 23:33:10.838030 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_b07cd9e36cc84081a4ac019dc2dcad7c [id=09ab2717b0c94e1caa955a381677e665]
I20260810 23:33:10.844776 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_90b34e5db74b40c1954de96c5c9fa3d1 [id=0bf66ae0183340f9b3394ad6b0ac5c79]
I20260810 23:33:10.853488 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_cd6a61f95ef74a6495c279e07bfd8b35 [id=0cbb4cd2d3f94123a65ebc79cd54a870]
I20260810 23:33:10.857191 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_496ebacf57f14572b620d213d8865579 [id=0e6874a7b48d40dd830fcdb02378f973]
I20260810 23:33:10.863816 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_a7072962772044ad8f3e9a8e446baea1 [id=103311fa8f874c1a9ce05417581d7483]
I20260810 23:33:10.870427 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_c244137f10db487f95395a64261b74b1 [id=12051526e7a441649a43d4edbd47a578]
I20260810 23:33:10.877745 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_023c0d2da11a419f8c2eb9dc2a064cee [id=1386100c13dc4d54975429c0dffbef72]
W20260810 23:33:10.879664 12215 catalog_manager.cc:4639] Async tablet task AlterTable RPC for tablet 0ff43927ed204bfda6f647808b44fa3a (table default.Table_dc4c377dd45047ba9e9c97d06a255575 [id=e539d34e2b7c48bebe9e2729946b3856], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0ff43927ed204bfda6f647808b44fa3a consensus state has no leader
I20260810 23:33:10.883221 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_83e4e8027f8b4f61b104e94ccf146481 [id=14bc67f80e344860827f60af309df4e8]
I20260810 23:33:10.886751 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_0bf7fdea09a840b9a1394935cc53d643 [id=15edc7d801a94b40b8ab324e25d530e4]
I20260810 23:33:10.888670 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_64b9e1ba04bf4003bc78a1f24cdbd715 [id=179a899b06c54cd888f52bf9ee665890]
I20260810 23:33:10.892324 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_7221fbab5f0b4fa892342870790a2f7e [id=1ea3dfab929845f89d4680129ae45d94]
I20260810 23:33:10.897553 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_9edef8525efa4d13af777efceb37a603 [id=225155f28cad4653997e017cdd21008a]
I20260810 23:33:10.899435 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_93dd9072dddd48ec859e82f2c6e0244c [id=24e3b49b5b8f4bcfabc485ddf55ee01f]
I20260810 23:33:10.901324 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_9a2539ffd1684a4cae73196a0ab97172 [id=254e99048db04a75ae17a9f262c8a0c0]
I20260810 23:33:10.904321 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_5b5e98d980b14439a4826410a95f81cb [id=265353c26473442fad44e2b973d46680]
I20260810 23:33:10.906332 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_241642ff9438427d8938dd88c19ba164 [id=2696163e827d4abca86cb624622710bd]
I20260810 23:33:10.908258 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_d134a55028664442b08b229593877973 [id=26be70827ca442a4b24a26015ffd952a]
I20260810 23:33:10.910065 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_c5751d9b450d4231a2d59983159349b7 [id=27425440b77a4601964b0eea5ba2fee1]
I20260810 23:33:10.912833 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_c608362e6ea74293b5e8a84421d98e10 [id=2be595c4bd574994afaa2e183e8c1a66]
I20260810 23:33:10.914686 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_4a7751c5b61c4332adeeb9ee00474b27 [id=2c8320255aab4036b5350d082f9e04ef]
I20260810 23:33:10.917531 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f]
I20260810 23:33:10.919288 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_4b640b4e6af94983b828197e5ece8d86 [id=32e343f820e2402d92a22104bf04b8ea]
I20260810 23:33:10.921167 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_cf212e8521974a72a6b86f2ea6ef2ffc [id=339a844c37ee460cb948d7e58264e5c8]
I20260810 23:33:10.923103 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_566f96b5dead41b78406bb2776c7121b [id=350cca934f6840188a31c37cb981e8f4]
I20260810 23:33:10.925091 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_a8f723dad5a54fceb62622c2aefe987c [id=35b68d151d0b47a5a90c0c4e1e762d74]
I20260810 23:33:10.927127 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_afb094287f0e4e88aefdd128d2fd7ef8 [id=39369f2619cf417ea917db1973d0270b]
I20260810 23:33:10.928980 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_34f40e29ad7a4068943dc4c724bce3b9 [id=3d5ee1f5b27f4707b79ed3fef7b8ffa4]
I20260810 23:33:10.930768 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_5e75f4eb90d9446d89a8395b2aa65a1d [id=3f2a52251dba491c8f6acfa4da4960e2]
I20260810 23:33:10.947376 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_535019edc4634bb68e6c59f1e8c56bac [id=47f3c40c679f4fc0b16afcb03946d8a0]
I20260810 23:33:10.951341 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_d028b6b163de469a9325b241cb5701a1 [id=49a3a44dd7f9473d9c025d8690a00c1f]
I20260810 23:33:10.953631 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_cd3600f5c033449dae5897454ce350a7 [id=4acb486be7454f6093439e084454ed36]
I20260810 23:33:10.963212 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_829882defbbb45f78cd9bb86c3f7303f [id=4eebce3cc6e346fb92e0cd5b62af2c50]
I20260810 23:33:10.995325 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_95b3b58c2c4948c199a9b573367fa598 [id=56bede34f9a1429daaed5c6077be5513]
I20260810 23:33:11.009773 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_763854d208924ff78919903a71a7c826 [id=575025603ae44b0785de76cc5a28ab9f]
I20260810 23:33:11.019315 15905 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:11.019903 15905 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:33:11.023527 15905 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:33:11.024586 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_02819bd703c44810832aecd72d795582 [id=59b578941afe4d26a2fa5e09137579b5]
I20260810 23:33:11.027544 15296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 12 candidate_status { last_received { term: 10 index: 447 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b" is_pre_election: true
I20260810 23:33:11.028287 15296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e964330cf113403a8aa5b73ddd3b31d0 in term 11.
I20260810 23:33:11.030861 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_0f0715c81643462985ef73882b0ccb39 [id=5b00eb54477e4461a5710c99f2517d85]
I20260810 23:33:11.032840 12215 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0; no voters:
I20260810 23:33:11.036850 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_c764bc5f83fe4b25aaff52cac5e1009e [id=5b473a1a3aff4ff9adfd4a0542cab809]
I20260810 23:33:11.037730 15905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260810 23:33:11.038184 15905 raft_consensus.cc:482] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:11.038607 15905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 11 FOLLOWER]: Advancing to term 12
I20260810 23:33:11.064168 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_a3a059d2601345d68eadf189772bcde5 [id=5cc3da1483184129834caaad3e509c24]
I20260810 23:33:11.071624 15905 raft_consensus.cc:504] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:33:11.076200 15905 leader_election.cc:288] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 12 election: Requested vote from peers 198f534261994047a68e1d509bb2fd8b (127.11.226.189:42101), 681674cb24274f10b200f264f155c2d1 (127.11.226.188:45477)
I20260810 23:33:11.078529 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_8371cab4a7ff44ceb3b02e4cb333fae1 [id=5f6c8ccff6664844be9afc52d4851118]
I20260810 23:33:11.081245 15296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e964330cf113403a8aa5b73ddd3b31d0" candidate_term: 12 candidate_status { last_received { term: 10 index: 447 } } ignore_live_leader: false dest_uuid: "198f534261994047a68e1d509bb2fd8b"
I20260810 23:33:11.081966 15296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 11 FOLLOWER]: Advancing to term 12
I20260810 23:33:11.084619 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_94d5b6058b6041b79fb4ac9e4e07f354 [id=5fd2e68f76b441b1854a99c3eaad5596]
I20260810 23:33:11.087464 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_767939c31e6e4afcaf3d29cef1e53985 [id=5fff4f293aaa42719e4cd0b3403b59ed]
I20260810 23:33:11.091317 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_98320d53af774eabb5d826d4c0732dad [id=652ca05bcdf547cdb178a2f5c91eaf03]
I20260810 23:33:11.093387 15296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e964330cf113403a8aa5b73ddd3b31d0 in term 12.
I20260810 23:33:11.098290 12215 leader_election.cc:302] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198f534261994047a68e1d509bb2fd8b, e964330cf113403a8aa5b73ddd3b31d0; no voters:
I20260810 23:33:11.099395 15905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 12 FOLLOWER]: Leader election won for term 12
I20260810 23:33:11.099985 15905 raft_consensus.cc:686] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [term 12 LEADER]: Becoming Leader. State: Replica: e964330cf113403a8aa5b73ddd3b31d0, State: Running, Role: LEADER
I20260810 23:33:11.101033 15905 consensus_queue.cc:237] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 447, Committed index: 447, Last appended: 10.447, Last appended by leader: 447, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:33:11.097621 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_714aaba90fb0449bb9a5cb6ac00bae8c [id=6a20c65432e44766a6572a26e1336116]
I20260810 23:33:11.108150 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_9a58bbb30a414d8b8ec6893b106ee0a1 [id=6b7c9683647d4c2e94418e9f2dcaa41e]
I20260810 23:33:11.121155 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_86a07768a7834d14812dbdd318a66e5a [id=6bd548ee039f4ae89acbfbc935b09c21]
I20260810 23:33:11.122646 15903 sys_catalog.cc:455] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e964330cf113403a8aa5b73ddd3b31d0. Latest consensus state: current_term: 12 leader_uuid: "e964330cf113403a8aa5b73ddd3b31d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } } }
I20260810 23:33:11.123322 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_399593e632684b20a3032bb34a182dec [id=7184e7fca1b54e72a7ba15d5453baa37]
I20260810 23:33:11.123641 15903 sys_catalog.cc:458] T 00000000000000000000000000000000 P e964330cf113403a8aa5b73ddd3b31d0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:11.127564 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_be4c5de03d7649a08296569cbe6a3503 [id=77ef566f33a94336b6f94c0a117312a1]
I20260810 23:33:11.136920 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_1de5219e712b481d8048407bd4cec52a [id=7cffa285f3a04f74a80ec7c12b7b3c5b]
I20260810 23:33:11.139382 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_69797cc483424ae99a99731b36130f0f [id=82e5367dce5143c7b396b145a3faccc5]
I20260810 23:33:11.141134 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_d239f0c3798a49c1bf1055dab20c108a [id=845a7c2ecd844114a10dd1f7fe03ba7e]
I20260810 23:33:11.156109 15914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:33:11.159176 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_b3a3530a31c5486b9502e2fbba173887 [id=84ef6dc51476443392a1adc3bc66c853]
I20260810 23:33:11.164003 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_b89242e803f549b8aca7b725f00cd775 [id=84fae17bae554c39a0c5135014b4ee36]
I20260810 23:33:11.171751 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_98ae6d61510e4c97bea03912bdbbe8cc [id=88256e2f65354f26b2cd7bb8f0b1b7fc]
I20260810 23:33:11.179134 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_4f9225f25cc842fbad1e9ce8c1dba105 [id=891bfbcb1df049deb88545fcc3fb6792]
I20260810 23:33:11.182703 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_7e1011c9a03343acb3bdf04b07eb962e [id=89c842ab64c546bbb967a2246a7220ac]
I20260810 23:33:11.187258 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_68bcc5970e834ef8bb810ac4f663c1d0 [id=956f3847e6384c07a3c61ee05e5ff9de]
I20260810 23:33:11.188928 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_970b3cb3161f4f5a9fec0f3dfa72f1a5 [id=963512beef0d4063b8e6a5738abd1d92]
I20260810 23:33:11.196053 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_764f15b83d234a90886ebfae787847ad [id=985f8cd96a404f6d8d322ed7f10f968b]
I20260810 23:33:11.209101 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_bf4e391ee20b45d393a269203b401bc0 [id=98b0f9ff8bb1428a9e1f9bef95d8e60c]
I20260810 23:33:11.211158 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_7f6d5cd3228d430f9e4a4376d055f8d0 [id=997d599555454bbca33e0b1d98f2f88b]
I20260810 23:33:11.213249 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_88ec9ab54a584cd996eac5baed70b544 [id=9a3df87ccec0410faeee4aed54a34357]
I20260810 23:33:11.214946 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_98f0ace30b644009a7f4cd8cbe85217c [id=9c3ca59291864ec5ae5bc902a81c2110]
W20260810 23:33:11.216046 12215 connection.cc:535] server connection from 127.11.226.131:49515 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:33:11.216629 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_91daf907141c4d738a6e3d655692427c [id=9d36464c8de043e7a91673f011908f50]
I20260810 23:33:11.230194 12170 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu with pid 12188
I20260810 23:33:11.238795 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_5e1f708104aa41118f94a7e35cecf800 [id=9e4c719f0c7a4f839aed37bc54648a36]
I20260810 23:33:11.240797 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_f1056e40bbd0468985b81347da8caef7 [id=9e7f38ba26c14a66bc36404e70351094]
I20260810 23:33:11.242564 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_adde4ae3b1394291be75cc80db4f457d [id=a02006db831d417597d63c0e7c9cad8c]
I20260810 23:33:11.245379 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_b48833820bf944b688ae9585fb4daf68 [id=a6914cfeca1743cd84151fed87a4acee]
I20260810 23:33:11.247229 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_4ad6c4068abe4c64a425bea91494c59a [id=a6ac59b4e35043e1b236644dca9a5c65]
I20260810 23:33:11.251384 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_b138f739a5db4c59af5e8e17ae77a665 [id=abbede238d2641f396da52ac62b64f3f]
I20260810 23:33:11.254347 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_ebe848b2f9ba4077a556369c87aa436a [id=ae7b5faec4c34d35981e0a1d464202ef]
I20260810 23:33:11.255877 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_ac935ba0ebf44788b4f41765bc914fe9 [id=aeb6bf538aa84a7fadd38df9ddb652c5]
I20260810 23:33:11.258469 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_c28d62ee4c2a432b82a28cc0111c4e4d [id=af6bc38dfa954733a32b98d5543a3ffc]
I20260810 23:33:11.261865 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_b78ae79905d8448085a6e318220c1584 [id=b0b48348e1b444bbb9ac2d2cdeb078d9]
I20260810 23:33:11.263548 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_dc485eb27eb64d70ba14404adc87fb9a [id=b27a53844fa3419d80a7313ba2437c1e]
I20260810 23:33:11.265573 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_1072812b5a8a49a1a6a1a94eb4e2b319 [id=b3a8515067ee4ce08f1d5dd8e72f56f2]
I20260810 23:33:11.267153 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_9cab5b899a3748378d28d31c84e8b090 [id=b582cc38f8f04c8f989139034482272b]
I20260810 23:33:11.268659 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_8a358dec4da84beda7fdf1eb485eb587 [id=b5cc65e503f7413fbade0c94e2e74b14]
I20260810 23:33:11.270761 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_02075f5963604d95b8d962f1071ebe64 [id=b8a33141d6c648649115fa6daff6d13e]
I20260810 23:33:11.272147 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_e156205c664d4b45b29a0e258e39cf34 [id=ba18db75ae0b49b4ac6246e838459efe]
I20260810 23:33:11.282781 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_a7d468b7f476404c8b400f3923cd1543 [id=bc138fc988a942b093d18a66e0cf605c]
I20260810 23:33:11.300576 15296 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 198f534261994047a68e1d509bb2fd8b [term 12 FOLLOWER]: Rejecting Update request from peer 681674cb24274f10b200f264f155c2d1 for earlier term 11. Current term is 12. Ops: []
I20260810 23:33:11.303570 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_13d7888530b048f38e069ee640677b1c [id=bc3dacd88f7e4fa78fbefdb197df991b]
I20260810 23:33:11.305454 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_4884ea66b39f44e99d957b62d1f38e0c [id=bc5681871c97482881fcbb19b32cf6f4]
I20260810 23:33:11.307973 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_20b50818a91248faac361d538df4fb30 [id=bdfbe648345f4e4e8f67c244f4a15f17]
I20260810 23:33:11.307152 15922 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 }, Status: INVALID_TERM, Last received: 0.0, Next index: 448, Last known committed idx: 371, Time since last communication: 0.001s
I20260810 23:33:11.309018 15915 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 11 LEADER]: Stepping down as leader of term 11
I20260810 23:33:11.309370 15915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 681674cb24274f10b200f264f155c2d1, State: Running, Role: LEADER
I20260810 23:33:11.310274 15915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 447, Committed index: 447, Last appended: 11.448, Last appended by leader: 448, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e964330cf113403a8aa5b73ddd3b31d0" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 43533 } } peers { permanent_uuid: "198f534261994047a68e1d509bb2fd8b" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 42101 } } peers { permanent_uuid: "681674cb24274f10b200f264f155c2d1" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 45477 } }
I20260810 23:33:11.311596 15915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681674cb24274f10b200f264f155c2d1 [term 11 FOLLOWER]: Advancing to term 12
I20260810 23:33:11.321142 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_32d9165e885d480881ced6b879bcd759 [id=bece3c4a5e8e4dc4bb23129238561321]
I20260810 23:33:11.322981 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_e1ff8765e6e547fda12a138edb0a5bb1 [id=c0859eb309eb48f2a1976147213b0aab]
I20260810 23:33:11.324857 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_730f4da35e68448585507866c203a391 [id=c48d5d1f759b42f3a5c5a6c64588b30f]
I20260810 23:33:11.327258 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_3b251f8589be43f29704b8906d49f6cd [id=c56022cc8cad4c7cbefafad0782cefde]
I20260810 23:33:11.331991 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_2e4541bbe3aa45c1a1c85533c2ed79f3 [id=c9327e173e574f75b4886311e0933acc]
I20260810 23:33:11.345958 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_cb78723192e84474b32e919ec39a002b [id=c99a6f0c231f49da9e354014bb4a3544]
I20260810 23:33:11.347549 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_abf0a10256d44150b985a6d480568080 [id=ca29798e86b540b3aea30c9a32f4e4c4]
I20260810 23:33:11.362777 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_eb8f4b9d6a8d4c9891bff8243880751d [id=d207f6aad69f4e269854886a771760c9]
I20260810 23:33:11.365819 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_9625301c8cda460981e80e753c785d4b [id=d79b564767e24f3b947e21b4e56e2e3b]
I20260810 23:33:11.368250 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_76308997ea5c4140ac84b26ccc7cc1bd [id=d90642b9034b48079d0c57fa25dd589e]
I20260810 23:33:11.369663 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_d36db682741e4ef6bde7416df55ab535 [id=da7b0a1b8cbc4acabf7fd986c43fcbf3]
I20260810 23:33:11.392805 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_f07540d9f70a42a99acf3b3613c98d18 [id=ddf98dc81008484db03e1d98df5634c2]
I20260810 23:33:11.394711 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_d9fd2a26e54c483ebbc082b5a4ce014f [id=deb4795f344f4f108092842f7bd08934]
I20260810 23:33:11.396404 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_72495aefc75943eeb8a43dd95ba2b03d [id=e10ad146e4414a6aa58eb11999d5ed27]
I20260810 23:33:11.397792 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_6b8e1ec40cdc4eda823368ca59437332 [id=e2df41b29ef8482b9ad579c172717ffb]
I20260810 23:33:11.399914 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_dc4c377dd45047ba9e9c97d06a255575 [id=e539d34e2b7c48bebe9e2729946b3856]
I20260810 23:33:11.401472 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_07f851c929404a589a8572028206f242 [id=e83a47cd23b54a369d16be6e29335407]
I20260810 23:33:11.402843 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_523cb26bd2ea40f1b341060ff15317d3 [id=ef51c5adb1694c948f0430dfd66c967d]
I20260810 23:33:11.404138 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_7b45b2e7e9c242039e2f372c0491fdf7 [id=f2d145b522f94058b2ca8c0491c13eb0]
I20260810 23:33:11.405380 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_3ae548119a244f178ea7e8222ab60395 [id=f315e4339efb4e61b98a5923ca97c823]
I20260810 23:33:11.420522 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_d54c3da29f874f2e83e5a10727950256 [id=f775b2636c034546ba65c9fc32fefecc]
I20260810 23:33:11.422062 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_e1e8440f714f4d55a1622ec3b2f537b6 [id=f7cc6d155f6a451e9f7109d7be8d8f83]
I20260810 23:33:11.424067 15909 catalog_manager.cc:669] Loaded metadata for table default.Table_78e8fac0b43b408abf61639343c2e8e8 [id=f8efd2f66f3c4968932a2cf4f70a7154]
W20260810 23:33:11.437516 12456 connection.cc:535] client connection to 127.11.226.190:43533 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:11.438086 12456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 2 similar messages]
I20260810 23:33:11.460269 12170 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu with pid 13549
I20260810 23:33:11.510828 15909 tablet_loader.cc:96] loaded metadata for tablet 00b1e631390b4b69a6ba8c7b1ba5cfef (table default.Table_abf0a10256d44150b985a6d480568080 [id=ca29798e86b540b3aea30c9a32f4e4c4])
I20260810 23:33:11.513613 15909 tablet_loader.cc:96] loaded metadata for tablet 00f68708a0b74a58b69581e8324a33f6 (table default.Table_d134a55028664442b08b229593877973 [id=26be70827ca442a4b24a26015ffd952a])
I20260810 23:33:11.515627 15909 tablet_loader.cc:96] loaded metadata for tablet 046f1a7983ec40489edeaba4d2595b1c (table default.Table_8a358dec4da84beda7fdf1eb485eb587 [id=b5cc65e503f7413fbade0c94e2e74b14])
I20260810 23:33:11.517279 15909 tablet_loader.cc:96] loaded metadata for tablet 0703cf71490c4c71a95f55863583a03c (table default.Table_4f38414090ae4a5cac18c58cccb303e4 [id=326b0d80eb584c2b904f2167629dc30f])
I20260810 23:33:11.518710 15909 tablet_loader.cc:96] loaded metadata for tablet 0703e531759b4709bd3917cd9d307dd2 (table default.Table_95b3b58c2c4948c199a9b573367fa598 [id=56bede34f9a1429daaed5c6077be5513])
I20260810 23:33:11.522300 15909 tablet_loader.cc:96] loaded metadata for tablet 09bdc7c2aace45da9fce81c4904db5d4 (table default.Table_2e4541bbe3aa45c1a1c85533c2ed79f3 [id=c9327e173e574f75b4886311e0933acc])
I20260810 23:33:11.523836 15909 tablet_loader.cc:96] loaded metadata for tablet 0a0136a1e890436fae572b46e64750e7 (table default.Table_32d9165e885d480881ced6b879bcd759 [id=bece3c4a5e8e4dc4bb23129238561321])
I20260810 23:33:11.525534 15909 tablet_loader.cc:96] loaded metadata for tablet 0bd999bbcc04431f826ffccf360fc56d (table default.Table_be4c5de03d7649a08296569cbe6a3503 [id=77ef566f33a94336b6f94c0a117312a1])
I20260810 23:33:11.529325 15909 tablet_loader.cc:96] loaded metadata for tablet 0d1b15dd545c4598aa5c7a9dc53618ec (table default.Table_0bf7fdea09a840b9a1394935cc53d643 [id=15edc7d801a94b40b8ab324e25d530e4])
I20260810 23:33:11.532771 15909 tablet_loader.cc:96] loaded metadata for tablet 0ff43927ed204bfda6f647808b44fa3a (table default.Table_dc4c377dd45047ba9e9c97d06a255575 [id=e539d34e2b7c48bebe9e2729946b3856])
I20260810 23:33:11.537163 15909 tablet_loader.cc:96] loaded metadata for tablet 12b8279474454c73ac935a7ccd004407 (table default.Table_a8f723dad5a54fceb62622c2aefe987c [id=35b68d151d0b47a5a90c0c4e1e762d74])
I20260810 23:33:11.539533 15909 tablet_loader.cc:96] loaded metadata for tablet 136f5a8af8ae46cdb3dce5344fce59ad (table default.Table_cb78723192e84474b32e919ec39a002b [id=c99a6f0c231f49da9e354014bb4a3544])
I20260810 23:33:11.542212 15909 tablet_loader.cc:96] loaded metadata for tablet 1432c9f43cd34983a5a072b9c800da87 (table default.Table_c28d62ee4c2a432b82a28cc0111c4e4d [id=af6bc38dfa954733a32b98d5543a3ffc])
I20260810 23:33:11.544916 15909 tablet_loader.cc:96] loaded metadata for tablet 182f4889e7484e989dec9bcbfc297bb9 (table default.Table_8371cab4a7ff44ceb3b02e4cb333fae1 [id=5f6c8ccff6664844be9afc52d4851118])
I20260810 23:33:11.547907 15909 tablet_loader.cc:96] loaded metadata for tablet 19f5d509a67741fe97dd041b2dda9d75 (table default.Table_94d5b6058b6041b79fb4ac9e4e07f354 [id=5fd2e68f76b441b1854a99c3eaad5596])
I20260810 23:33:11.552783 15909 tablet_loader.cc:96] loaded metadata for tablet 217a7582902244f9b0ca65be1c20af22 (table default.Table_d028b6b163de469a9325b241cb5701a1 [id=49a3a44dd7f9473d9c025d8690a00c1f])
I20260810 23:33:11.556329 15909 tablet_loader.cc:96] loaded metadata for tablet 27c39d751e544d9a8afa26dc935a819a (table default.Table_78e8fac0b43b408abf61639343c2e8e8 [id=f8efd2f66f3c4968932a2cf4f70a7154])
I20260810 23:33:11.557993 15909 tablet_loader.cc:96] loaded metadata for tablet 296330391f7e4af99675bc6d7fded8b8 (table default.Table_f07540d9f70a42a99acf3b3613c98d18 [id=ddf98dc81008484db03e1d98df5634c2])
I20260810 23:33:11.559791 15909 tablet_loader.cc:96] loaded metadata for tablet 2bf4ff4b1264442b8ca1dbde64583220 (table default.Table_e156205c664d4b45b29a0e258e39cf34 [id=ba18db75ae0b49b4ac6246e838459efe])
I20260810 23:33:11.562407 15909 tablet_loader.cc:96] loaded metadata for tablet 2cfe8fa607664ee8a5403eabe8f844b9 (table default.Table_b07cd9e36cc84081a4ac019dc2dcad7c [id=09ab2717b0c94e1caa955a381677e665])
I20260810 23:33:11.564967 15909 tablet_loader.cc:96] loaded metadata for tablet 3127a5a12a9a433690dc5f8b0d474083 (table default.Table_829882defbbb45f78cd9bb86c3f7303f [id=4eebce3cc6e346fb92e0cd5b62af2c50])
I20260810 23:33:11.566745 15909 tablet_loader.cc:96] loaded metadata for tablet 31730595c84a44579c4e9db09ef603ba (table default.Table_6b8e1ec40cdc4eda823368ca59437332 [id=e2df41b29ef8482b9ad579c172717ffb])
I20260810 23:33:11.569656 15909 tablet_loader.cc:96] loaded metadata for tablet 35148d28de80407c954839099623e817 (table default.Table_a7d468b7f476404c8b400f3923cd1543 [id=bc138fc988a942b093d18a66e0cf605c])
I20260810 23:33:11.571353 15909 tablet_loader.cc:96] loaded metadata for tablet 39ff3bb1947c4f689353587396d878b8 (table default.Table_e1e8440f714f4d55a1622ec3b2f537b6 [id=f7cc6d155f6a451e9f7109d7be8d8f83])
I20260810 23:33:11.573098 15909 tablet_loader.cc:96] loaded metadata for tablet 3a85273186994f2885a82ef2fe020ca4 (table default.Table_d54c3da29f874f2e83e5a10727950256 [id=f775b2636c034546ba65c9fc32fefecc])
I20260810 23:33:11.577543 15909 tablet_loader.cc:96] loaded metadata for tablet 3fef127263f1470ea1056a454971e465 (table default.Table_023c0d2da11a419f8c2eb9dc2a064cee [id=1386100c13dc4d54975429c0dffbef72])
I20260810 23:33:11.579095 15909 tablet_loader.cc:96] loaded metadata for tablet 403caf051714438bb77fc2fbefca15c5 (table default.Table_68bcc5970e834ef8bb810ac4f663c1d0 [id=956f3847e6384c07a3c61ee05e5ff9de])
I20260810 23:33:11.581612 15909 tablet_loader.cc:96] loaded metadata for tablet 4166ff56ed074eca95dd909f36a8a28b (table default.Table_535019edc4634bb68e6c59f1e8c56bac [id=47f3c40c679f4fc0b16afcb03946d8a0])
I20260810 23:33:11.585916 15909 tablet_loader.cc:96] loaded metadata for tablet 427fa7e750e24ed9a48988fd3cb4ce1f (table default.Table_76308997ea5c4140ac84b26ccc7cc1bd [id=d90642b9034b48079d0c57fa25dd589e])
I20260810 23:33:11.589223 15909 tablet_loader.cc:96] loaded metadata for tablet 45e39a7b58ac402da7c5e845ea6c8329 (table default.Table_eb8f4b9d6a8d4c9891bff8243880751d [id=d207f6aad69f4e269854886a771760c9])
I20260810 23:33:11.592206 15909 tablet_loader.cc:96] loaded metadata for tablet 474a6165ce1440a294c4878602f0aed3 (table default.Table_ac935ba0ebf44788b4f41765bc914fe9 [id=aeb6bf538aa84a7fadd38df9ddb652c5])
I20260810 23:33:11.593333 12170 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu with pid 12410
2026-08-10T23:33:11Z chronyd exiting
I20260810 23:33:11.701193 12170 master-stress-test.cc:586] Tables created: 178
I20260810 23:33:11.701433 12170 master-stress-test.cc:587] Tables altered: 82
I20260810 23:33:11.701543 12170 master-stress-test.cc:588] Tables deleted: 69
I20260810 23:33:11.701666 12170 master-stress-test.cc:589] Tablets replaced: 33
I20260810 23:33:11.701788 12170 master-stress-test.cc:590] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (69418 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
I20260810 23:33:11.810427 12170 test_util.cc:268] Using random seed: -955614894
2026-08-10T23:33:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:33:11Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskt0RZpf/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskt0RZpf/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/security.properties
2026-08-10 23:33:20: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskt0RZpf/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskt0RZpf/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260810 23:33:31.635360 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.190:42189
--webserver_interface=127.11.226.190
--webserver_port=0
--builtin_ntp_servers=127.11.226.148:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--hive_metastore_uris=thrift://127.0.0.1:41643
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260810 23:33:32.025594 16285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:32.026403 16285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:32.026955 16285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:32.063014 16285 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260810 23:33:32.063514 16285 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260810 23:33:32.063858 16285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:33:32.064188 16285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:32.064512 16285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:33:32.064805 16285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:33:32.106976 16285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41643
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.190:42189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.226.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:33:32.108807 16285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:32.110965 16285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:32.136936 16298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:32.137339 16293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:32.136953 16294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:33.437443 16296 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 23:33:33.437628 16285 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 23:33:33.442915 16285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:33.447137 16285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:33.448704 16285 hybrid_clock.cc:648] HybridClock initialized: now 1786404813448662 us; error 78 us; skew 500 ppm
I20260810 23:33:33.449676 16285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:33.463428 16285 webserver.cc:460] Webserver started at http://127.11.226.190:35141/ using document root <none> and password file <none>
I20260810 23:33:33.464795 16285 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:33.465052 16285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:33.465576 16285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:33.472440 16285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/data/instance:
uuid: "61534c7c271c4e3393ff5b46a3498f9b"
format_stamp: "Formatted at 2026-08-10 23:33:33 on dist-test-slave-kvfs"
I20260810 23:33:33.474063 16285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/wal/instance:
uuid: "61534c7c271c4e3393ff5b46a3498f9b"
format_stamp: "Formatted at 2026-08-10 23:33:33 on dist-test-slave-kvfs"
I20260810 23:33:33.484256 16285 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.005s
I20260810 23:33:33.493513 16310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:33.495256 16285 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260810 23:33:33.495733 16285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/wal
uuid: "61534c7c271c4e3393ff5b46a3498f9b"
format_stamp: "Formatted at 2026-08-10 23:33:33 on dist-test-slave-kvfs"
I20260810 23:33:33.496408 16285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:33.540397 16285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:33.542363 16285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:33.542934 16285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:33.639463 16285 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.190:42189
I20260810 23:33:33.639600 16385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.190:42189 every 8 connection(s)
I20260810 23:33:33.643110 16285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/data/info.pb
I20260810 23:33:33.649799 16387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:33.653112 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 16285
I20260810 23:33:33.653934 12170 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-0/wal/instance
I20260810 23:33:33.657305 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.189:34939
--webserver_interface=127.11.226.189
--webserver_port=0
--builtin_ntp_servers=127.11.226.148:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--hive_metastore_uris=thrift://127.0.0.1:41643
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260810 23:33:33.664573 16387 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:33:33.703657 16387 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } has no permanent_uuid. Determining permanent_uuid...
W20260810 23:33:33.706610 16314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
W20260810 23:33:33.712607 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:34939: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:33:33.762327 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } attempt: 1
W20260810 23:33:33.767127 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:34939: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:33:33.835713 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } attempt: 2
W20260810 23:33:33.840939 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:34939: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:33:33.932595 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } attempt: 3
W20260810 23:33:33.937562 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:34939: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
W20260810 23:33:34.057183 16391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:34.057910 16391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:34.058367 16391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:33:34.081209 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } attempt: 4
W20260810 23:33:34.085934 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:34939: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
W20260810 23:33:34.094861 16391 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260810 23:33:34.095171 16391 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260810 23:33:34.095381 16391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:33:34.095583 16391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:34.095781 16391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:33:34.095994 16391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:33:34.140210 16391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41643
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.189:34939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.226.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:33:34.142066 16391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:34.144750 16391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:34.169296 16414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:34.385548 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } attempt: 5
W20260810 23:33:34.391583 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:34939: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:33:34.939163 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } attempt: 6
W20260810 23:33:34.948238 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.189:34939: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
W20260810 23:33:34.169320 16409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:34.169721 16407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:35.448079 16412 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 23:33:35.448406 16391 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 23:33:35.453756 16391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:35.458007 16391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:35.459635 16391 hybrid_clock.cc:648] HybridClock initialized: now 1786404815459578 us; error 86 us; skew 500 ppm
I20260810 23:33:35.460764 16391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:35.469645 16391 webserver.cc:460] Webserver started at http://127.11.226.189:39779/ using document root <none> and password file <none>
I20260810 23:33:35.470953 16391 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:35.471213 16391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:35.471760 16391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:35.478253 16391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data/instance:
uuid: "d0596cdb0a514cc08d7df0ceea528339"
format_stamp: "Formatted at 2026-08-10 23:33:35 on dist-test-slave-kvfs"
I20260810 23:33:35.479617 16391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/wal/instance:
uuid: "d0596cdb0a514cc08d7df0ceea528339"
format_stamp: "Formatted at 2026-08-10 23:33:35 on dist-test-slave-kvfs"
I20260810 23:33:35.489053 16391 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260810 23:33:35.496624 16426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:35.498157 16391 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260810 23:33:35.498533 16391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/wal
uuid: "d0596cdb0a514cc08d7df0ceea528339"
format_stamp: "Formatted at 2026-08-10 23:33:35 on dist-test-slave-kvfs"
I20260810 23:33:35.498942 16391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:35.548712 16391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:35.550675 16391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:35.551222 16391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:35.652882 16391 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.189:34939
I20260810 23:33:35.652890 16501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.189:34939 every 8 connection(s)
I20260810 23:33:35.657807 16391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data/info.pb
I20260810 23:33:35.658514 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 16391
I20260810 23:33:35.659078 12170 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/wal/instance
I20260810 23:33:35.663704 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.188:46109
--webserver_interface=127.11.226.188
--webserver_port=0
--builtin_ntp_servers=127.11.226.148:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--hive_metastore_uris=thrift://127.0.0.1:41643
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260810 23:33:35.666342 16502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:35.687033 16502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:33:35.724190 16502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:33:35.744675 16502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } has no permanent_uuid. Determining permanent_uuid...
W20260810 23:33:35.747675 16435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
W20260810 23:33:35.753403 16502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
I20260810 23:33:35.803177 16502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 1
W20260810 23:33:35.807902 16502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
I20260810 23:33:35.876533 16502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 2
W20260810 23:33:35.881560 16502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
I20260810 23:33:35.973129 16502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 3
W20260810 23:33:35.978106 16502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
I20260810 23:33:36.009171 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } attempt: 7
I20260810 23:33:36.026297 16387 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } has no permanent_uuid. Determining permanent_uuid...
W20260810 23:33:36.034219 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
W20260810 23:33:36.090232 16504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:36.091079 16504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:36.091635 16504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:33:36.092897 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 1
W20260810 23:33:36.097642 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
I20260810 23:33:36.121637 16502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 4
W20260810 23:33:36.126585 16502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
W20260810 23:33:36.132923 16504 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260810 23:33:36.133294 16504 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260810 23:33:36.133590 16504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:33:36.133813 16504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:36.134032 16504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:33:36.134231 16504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:33:36.177492 16504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41643
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.188:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.226.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:33:36.179163 16504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:36.179171 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 2
I20260810 23:33:36.182216 16504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:36.184265 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
W20260810 23:33:36.208441 16532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:36.269739 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 3
W20260810 23:33:36.276068 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
I20260810 23:33:36.416684 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 4
W20260810 23:33:36.422412 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
I20260810 23:33:36.426146 16502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 5
W20260810 23:33:36.431957 16502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
I20260810 23:33:36.706044 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 5
W20260810 23:33:36.711043 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
I20260810 23:33:36.979511 16502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 6
W20260810 23:33:36.989190 16502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
I20260810 23:33:37.263928 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 6
W20260810 23:33:37.272832 16387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.226.188:46109: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
W20260810 23:33:36.208467 16526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:36.208976 16529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:37.513578 16531 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 23:33:37.513764 16504 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 23:33:37.520187 16504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:37.524631 16504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:37.526099 16504 hybrid_clock.cc:648] HybridClock initialized: now 1786404817526052 us; error 84 us; skew 500 ppm
I20260810 23:33:37.527225 16504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:37.536120 16504 webserver.cc:460] Webserver started at http://127.11.226.188:46275/ using document root <none> and password file <none>
I20260810 23:33:37.537475 16504 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:37.537746 16504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:37.538393 16504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:37.545047 16504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data/instance:
uuid: "0bfd786d28e24be390cdbbb4007290a7"
format_stamp: "Formatted at 2026-08-10 23:33:37 on dist-test-slave-kvfs"
I20260810 23:33:37.546597 16504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/wal/instance:
uuid: "0bfd786d28e24be390cdbbb4007290a7"
format_stamp: "Formatted at 2026-08-10 23:33:37 on dist-test-slave-kvfs"
I20260810 23:33:37.556815 16504 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20260810 23:33:37.564167 16552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:37.565687 16504 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:33:37.566200 16504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/wal
uuid: "0bfd786d28e24be390cdbbb4007290a7"
format_stamp: "Formatted at 2026-08-10 23:33:37 on dist-test-slave-kvfs"
I20260810 23:33:37.566623 16504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:37.607275 16504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:37.609325 16504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:37.609901 16504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:37.706097 16504 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.188:46109
I20260810 23:33:37.706192 16624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.188:46109 every 8 connection(s)
I20260810 23:33:37.709539 16504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data/info.pb
I20260810 23:33:37.710273 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 16504
I20260810 23:33:37.710870 12170 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/wal/instance
I20260810 23:33:37.717664 16625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:37.738318 16625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:33:37.769694 16625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:33:37.783668 16625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:33:37.804710 16625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: Bootstrap starting.
I20260810 23:33:37.813516 16625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:37.815927 16625 log.cc:824] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:37.822943 16625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: No bootstrap required, opened a new log
I20260810 23:33:37.846894 16625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:37.847805 16625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:37.848219 16625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bfd786d28e24be390cdbbb4007290a7, State: Initialized, Role: FOLLOWER
I20260810 23:33:37.849371 16625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:37.852936 16636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:37.853940 16636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:37.854661 16625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:33:37.866446 16562 tablet.cc:2369] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:33:37.895557 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:37.896248 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:33:37.912993 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:37.913638 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:37.889348 16504 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: OK, instance_id { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" instance_seqno: 1786404813612732 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:33:37.929963 16504 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:33:37.936193 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:37.936872 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:33:37.954886 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:37.955451 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:33:37.970849 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:37.971367 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:33:37.985687 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:37.986040 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:33:38.004733 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.005211 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:33:38.019349 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.019989 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:33:38.033958 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.034323 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.049818 16502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 7
W20260810 23:33:38.050014 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 23:33:38.050520 16636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:33:38.051230 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.051589 16636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:38.054780 16636 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:33:38.066232 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.066752 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.081223 16502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Bootstrap starting.
W20260810 23:33:38.081744 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.082115 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.090749 16502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:38.093420 16502 log.cc:824] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Log is configured to *not* fsync() on all Append() calls
W20260810 23:33:38.105504 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.105942 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.107419 16502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: No bootstrap required, opened a new log
I20260810 23:33:38.135617 16502 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
W20260810 23:33:38.135406 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 23:33:38.136533 16502 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:38.136897 16502 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0596cdb0a514cc08d7df0ceea528339, State: Initialized, Role: FOLLOWER
W20260810 23:33:38.136934 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.137938 16502 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:38.141387 16663 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:38.142442 16663 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:38.143586 16502 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:33:38.143874 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:38.145016 16468 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bfd786d28e24be390cdbbb4007290a7 in term 0.
I20260810 23:33:38.147153 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7, d0596cdb0a514cc08d7df0ceea528339; no voters:
I20260810 23:33:38.148773 16636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:38.149313 16636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:38.149734 16636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:33:38.153029 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.153414 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.165339 16636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:38.171826 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339"
I20260810 23:33:38.172788 16468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:38.182842 16468 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bfd786d28e24be390cdbbb4007290a7 in term 1.
I20260810 23:33:38.183648 16636 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 1 election: Requested vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
I20260810 23:33:38.185906 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7, d0596cdb0a514cc08d7df0ceea528339; no voters:
I20260810 23:33:38.187397 16636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 1 FOLLOWER]: Leader election won for term 1
W20260810 23:33:38.179839 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.189714 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.194694 16636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 1 LEADER]: Becoming Leader. State: Replica: 0bfd786d28e24be390cdbbb4007290a7, State: Running, Role: LEADER
I20260810 23:33:38.196395 16636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:38.215047 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bfd786d28e24be390cdbbb4007290a7. Latest consensus state: current_term: 1 leader_uuid: "0bfd786d28e24be390cdbbb4007290a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:38.216403 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: LEADER
W20260810 23:33:38.219978 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.221397 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:33:38.243017 16658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.244709 16658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.251925 16681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:33:38.270560 16681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:33:38.306371 16387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } attempt: 7
W20260810 23:33:38.311628 16437 tablet.cc:2369] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:33:38.316744 16693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.317241 16693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:33:38.333194 16693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.333645 16693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:33:38.348479 16693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.348863 16693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.349584 16468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 1 FOLLOWER]: Refusing update from remote peer 0bfd786d28e24be390cdbbb4007290a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:33:38.361910 16669 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:33:38.364681 16693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.365033 16693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.381676 16387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Bootstrap starting.
I20260810 23:33:38.391710 16387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:38.394989 16387 log.cc:824] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:38.426939 16387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: No bootstrap required, opened a new log
I20260810 23:33:38.292685 16391 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: OK, instance_id { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" instance_seqno: 1786404813612732 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:33:38.440410 16391 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:33:38.454778 16663 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bfd786d28e24be390cdbbb4007290a7. Latest consensus state: current_term: 1 leader_uuid: "0bfd786d28e24be390cdbbb4007290a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:38.456178 16663 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:38.472169 16698 mvcc.cc:205] Tried to move back new op lower bound from 7317114135938666496 to 7317114135348977664. Current Snapshot: MvccSnapshot[applied={T|T < 7317114135938666496}]
W20260810 23:33:38.476564 16693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.477032 16693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.485564 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bfd786d28e24be390cdbbb4007290a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:38.486678 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: LEADER
W20260810 23:33:38.494148 16693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:33:38.494525 16693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.494190 16663 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bfd786d28e24be390cdbbb4007290a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:38.496117 16663 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:38.499472 16681 catalog_manager.cc:1347] Generated new cluster ID: 82c629e15cdf4d9cb19224cd28fe2c13
I20260810 23:33:38.499852 16681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:33:38.515666 16387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:38.525624 16693 catalog_manager.cc:1259] Loaded cluster ID: 82c629e15cdf4d9cb19224cd28fe2c13
I20260810 23:33:38.522120 16387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:38.526748 16387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61534c7c271c4e3393ff5b46a3498f9b, State: Initialized, Role: FOLLOWER
I20260810 23:33:38.526587 16693 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: loading cluster ID for follower catalog manager: success
I20260810 23:33:38.528373 16387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
W20260810 23:33:38.542469 16693 catalog_manager.cc:1576] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260810 23:33:38.543114 16693 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.547758 16387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:33:38.548532 16350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:38.549176 16352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b" is_pre_election: true
I20260810 23:33:38.550084 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b"
W20260810 23:33:38.557703 16321 tablet.cc:2369] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:33:38.566756 16350 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 1 FOLLOWER]: Refusing update from remote peer 0bfd786d28e24be390cdbbb4007290a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 23:33:38.572827 16693 catalog_manager.cc:1576] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260810 23:33:38.573203 16693 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.577891 16636 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:33:38.583381 16705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:38.584669 16705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:38.594437 16681 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:33:38.599692 16681 catalog_manager.cc:1504] Loading token signing keys...
W20260810 23:33:38.604048 16693 catalog_manager.cc:1576] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260810 23:33:38.604631 16693 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.627837 16693 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: acquiring CA information for follower catalog manager: success
W20260810 23:33:38.633387 16693 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260810 23:33:38.668509 16681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: Generated new TSK 0
I20260810 23:33:38.670189 16681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:33:38.675532 16681 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260810 23:33:38.682387 16681 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260810 23:33:38.690905 16705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bfd786d28e24be390cdbbb4007290a7. Latest consensus state: current_term: 1 leader_uuid: "0bfd786d28e24be390cdbbb4007290a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:38.691783 16705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:38.694917 16724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bfd786d28e24be390cdbbb4007290a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:38.696076 16724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:38.687906 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bfd786d28e24be390cdbbb4007290a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:38.696557 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:38.714006 16693 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:33:38.719323 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.226.129:0
--local_ip_for_outbound_sockets=127.11.226.129
--webserver_interface=127.11.226.129
--webserver_port=0
--tserver_master_addrs=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--builtin_ntp_servers=127.11.226.148:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260810 23:33:38.757505 16734 catalog_manager.cc:1259] Loaded cluster ID: 82c629e15cdf4d9cb19224cd28fe2c13
I20260810 23:33:38.764221 16734 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: loading cluster ID for follower catalog manager: success
I20260810 23:33:38.790920 16734 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: acquiring CA information for follower catalog manager: success
I20260810 23:33:38.822172 16734 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 23:33:39.463991 16736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:39.464927 16736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:39.466190 16736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:39.536129 16736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:39.537977 16736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.226.129
I20260810 23:33:39.620368 16736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.226.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.226.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.226.129
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:33:39.623137 16736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:39.627113 16736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:39.690124 16746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:40.039083 16752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0bfd786d28e24be390cdbbb4007290a7)
I20260810 23:33:40.039939 16752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:40.054464 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:40.060043 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7" is_pre_election: true
I20260810 23:33:40.062507 16316 leader_election.cc:302] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61534c7c271c4e3393ff5b46a3498f9b; no voters: 0bfd786d28e24be390cdbbb4007290a7, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:40.071134 16753 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:33:40.071188 16752 leader_election.cc:288] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:40.100526 16754 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0bfd786d28e24be390cdbbb4007290a7)
I20260810 23:33:40.101517 16754 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:40.115593 16754 leader_election.cc:288] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:40.120734 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0596cdb0a514cc08d7df0ceea528339" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b" is_pre_election: true
I20260810 23:33:40.122658 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0596cdb0a514cc08d7df0ceea528339" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7" is_pre_election: true
I20260810 23:33:40.132475 16433 leader_election.cc:302] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0596cdb0a514cc08d7df0ceea528339; no voters: 0bfd786d28e24be390cdbbb4007290a7, 61534c7c271c4e3393ff5b46a3498f9b
I20260810 23:33:40.136590 16754 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:33:41.020526 16758 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0bfd786d28e24be390cdbbb4007290a7)
I20260810 23:33:41.021884 16758 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:41.028096 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0596cdb0a514cc08d7df0ceea528339" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b" is_pre_election: true
I20260810 23:33:41.033449 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0596cdb0a514cc08d7df0ceea528339" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7" is_pre_election: true
I20260810 23:33:41.036092 16435 leader_election.cc:302] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0596cdb0a514cc08d7df0ceea528339; no voters: 0bfd786d28e24be390cdbbb4007290a7, 61534c7c271c4e3393ff5b46a3498f9b
I20260810 23:33:41.042815 16758 leader_election.cc:288] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:41.044090 16758 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:33:41.103463 16761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0bfd786d28e24be390cdbbb4007290a7)
I20260810 23:33:41.108774 16761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:41.113597 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:41.114840 16468 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61534c7c271c4e3393ff5b46a3498f9b in term 1.
I20260810 23:33:41.116706 16314 leader_election.cc:302] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339; no voters:
I20260810 23:33:41.129230 16762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:33:41.132575 16762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0bfd786d28e24be390cdbbb4007290a7)
I20260810 23:33:41.132450 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7" is_pre_election: true
I20260810 23:33:41.133198 16762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:33:41.140216 16761 leader_election.cc:288] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:41.182895 16762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:41.186538 16351 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 2 FOLLOWER]: Rejecting Update request from peer 0bfd786d28e24be390cdbbb4007290a7 for earlier term 1. Current term is 2. Ops: []
I20260810 23:33:41.195248 16763 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 23:33:41.197103 16763 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 23:33:41.197635 16763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0bfd786d28e24be390cdbbb4007290a7, State: Running, Role: LEADER
I20260810 23:33:41.201025 16763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:41.205341 16763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:33:41.214361 16762 leader_election.cc:288] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 2 election: Requested vote from peers d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:41.217666 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339"
I20260810 23:33:41.219285 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7"
I20260810 23:33:41.269428 16591 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61534c7c271c4e3393ff5b46a3498f9b in term 2.
I20260810 23:33:41.270916 16658 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:33:41.282207 16316 leader_election.cc:302] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7, 61534c7c271c4e3393ff5b46a3498f9b; no voters: d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:41.283244 16762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:33:41.283811 16762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 2 LEADER]: Becoming Leader. State: Replica: 61534c7c271c4e3393ff5b46a3498f9b, State: Running, Role: LEADER
I20260810 23:33:41.284870 16762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:41.303506 16761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61534c7c271c4e3393ff5b46a3498f9b. Latest consensus state: current_term: 2 leader_uuid: "61534c7c271c4e3393ff5b46a3498f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:41.304919 16761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:41.316527 16767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:33:41.353224 16767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:33:41.374472 16767 catalog_manager.cc:1259] Loaded cluster ID: 82c629e15cdf4d9cb19224cd28fe2c13
I20260810 23:33:41.375329 16767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:33:41.395762 16767 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:33:41.412591 16767 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Loaded TSK: 0
I20260810 23:33:41.423457 16767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:33:41.434970 16767 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260810 23:33:41.458253 16767 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260810 23:33:41.651577 16769 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0bfd786d28e24be390cdbbb4007290a7)
I20260810 23:33:41.652509 16769 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:41.657166 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0596cdb0a514cc08d7df0ceea528339" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7" is_pre_election: true
I20260810 23:33:41.657207 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0596cdb0a514cc08d7df0ceea528339" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b" is_pre_election: true
I20260810 23:33:41.658201 16591 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0596cdb0a514cc08d7df0ceea528339 in current term 2: Already voted for candidate 61534c7c271c4e3393ff5b46a3498f9b in this term.
I20260810 23:33:41.660346 16435 leader_election.cc:302] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0596cdb0a514cc08d7df0ceea528339; no voters: 0bfd786d28e24be390cdbbb4007290a7, 61534c7c271c4e3393ff5b46a3498f9b
I20260810 23:33:41.673985 16769 leader_election.cc:288] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:41.675838 16771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:33:41.685410 16771 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:33:41.751639 16591 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 2 FOLLOWER]: Refusing update from remote peer 61534c7c271c4e3393ff5b46a3498f9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 23:33:41.754122 16761 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 23:33:41.799681 16772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61534c7c271c4e3393ff5b46a3498f9b. Latest consensus state: current_term: 2 leader_uuid: "61534c7c271c4e3393ff5b46a3498f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:41.800853 16772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:41.810417 16772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "61534c7c271c4e3393ff5b46a3498f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:41.811265 16772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:41.821650 16468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 2 FOLLOWER]: Refusing update from remote peer 61534c7c271c4e3393ff5b46a3498f9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 23:33:41.823668 16762 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260810 23:33:41.832924 16769 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "61534c7c271c4e3393ff5b46a3498f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:41.833804 16769 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:41.835060 16771 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61534c7c271c4e3393ff5b46a3498f9b. Latest consensus state: current_term: 2 leader_uuid: "61534c7c271c4e3393ff5b46a3498f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:41.835805 16771 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:41.866144 16761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "61534c7c271c4e3393ff5b46a3498f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:41.867293 16761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:41.868306 16762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "61534c7c271c4e3393ff5b46a3498f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:41.869033 16762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:42.308156 16772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61534c7c271c4e3393ff5b46a3498f9b)
I20260810 23:33:42.308888 16772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:42.315910 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b" is_pre_election: true
I20260810 23:33:42.319880 16772 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
I20260810 23:33:42.321697 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:42.324607 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:42.326476 16772 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 23:33:42.327901 16771 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61534c7c271c4e3393ff5b46a3498f9b)
I20260810 23:33:42.328476 16771 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:42.330718 16771 leader_election.cc:288] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:42.332599 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0596cdb0a514cc08d7df0ceea528339" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7" is_pre_election: true
I20260810 23:33:42.332959 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0596cdb0a514cc08d7df0ceea528339" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b" is_pre_election: true
I20260810 23:33:42.333588 16591 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0596cdb0a514cc08d7df0ceea528339 in term 2.
I20260810 23:33:42.336637 16435 leader_election.cc:302] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7, d0596cdb0a514cc08d7df0ceea528339; no voters: 61534c7c271c4e3393ff5b46a3498f9b
I20260810 23:33:42.338246 16771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 23:33:42.338604 16771 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61534c7c271c4e3393ff5b46a3498f9b)
I20260810 23:33:42.340010 16771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:33:42.358714 16771 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
W20260810 23:33:39.700755 16749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:42.365226 16736 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260810 23:33:42.366154 16747 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 23:33:42.368245 16468 raft_consensus.cc:1226] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 3 FOLLOWER]: Rejecting Update request from peer 61534c7c271c4e3393ff5b46a3498f9b for earlier term 2. Current term is 3. Ops: []
I20260810 23:33:42.370662 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0596cdb0a514cc08d7df0ceea528339" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b"
I20260810 23:33:42.371052 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0596cdb0a514cc08d7df0ceea528339" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7"
I20260810 23:33:42.371752 16591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:33:42.372179 16761 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 23:33:42.373466 16761 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 2 LEADER]: Stepping down as leader of term 2
I20260810 23:33:42.373926 16761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61534c7c271c4e3393ff5b46a3498f9b, State: Running, Role: LEADER
W20260810 23:33:39.692179 16744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:42.375824 16761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:42.377429 16736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:42.377705 16761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:33:42.380403 16591 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0596cdb0a514cc08d7df0ceea528339 in term 3.
I20260810 23:33:42.381026 16771 leader_election.cc:288] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 3 election: Requested vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:42.384125 16435 leader_election.cc:302] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7, d0596cdb0a514cc08d7df0ceea528339; no voters: 61534c7c271c4e3393ff5b46a3498f9b
I20260810 23:33:42.385638 16771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:33:42.392232 16771 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 3 LEADER]: Becoming Leader. State: Replica: d0596cdb0a514cc08d7df0ceea528339, State: Running, Role: LEADER
I20260810 23:33:42.391391 16736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:42.394608 16771 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:42.400708 16736 hybrid_clock.cc:648] HybridClock initialized: now 1786404822400570 us; error 156 us; skew 500 ppm
I20260810 23:33:42.402170 16736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:42.402171 16786 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0596cdb0a514cc08d7df0ceea528339. Latest consensus state: current_term: 3 leader_uuid: "d0596cdb0a514cc08d7df0ceea528339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:42.403118 16786 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:42.415797 16788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:33:42.434275 16788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:33:42.439201 16788 catalog_manager.cc:1259] Loaded cluster ID: 82c629e15cdf4d9cb19224cd28fe2c13
I20260810 23:33:42.439565 16788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:33:42.460623 16788 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:33:42.461026 16736 webserver.cc:460] Webserver started at http://127.11.226.129:41661/ using document root <none> and password file <none>
I20260810 23:33:42.462756 16736 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:42.463107 16736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:42.463863 16736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:42.474277 16736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/data/instance:
uuid: "c3b587fa042b42e4b66bb113245c0e36"
format_stamp: "Formatted at 2026-08-10 23:33:42 on dist-test-slave-kvfs"
I20260810 23:33:42.466148 16788 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Loaded TSK: 0
I20260810 23:33:42.476509 16736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/instance:
uuid: "c3b587fa042b42e4b66bb113245c0e36"
format_stamp: "Formatted at 2026-08-10 23:33:42 on dist-test-slave-kvfs"
I20260810 23:33:42.477994 16788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:33:42.482903 16788 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260810 23:33:42.487470 16788 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260810 23:33:42.495908 16736 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.011s sys 0.004s
I20260810 23:33:42.511835 16793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:42.513773 16736 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.009s sys 0.000s
I20260810 23:33:42.514315 16736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal
uuid: "c3b587fa042b42e4b66bb113245c0e36"
format_stamp: "Formatted at 2026-08-10 23:33:42 on dist-test-slave-kvfs"
I20260810 23:33:42.514907 16736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:42.604367 16736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:42.606482 16736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:42.607169 16736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:42.612627 16736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:33:42.620096 16736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:33:42.620487 16736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 23:33:42.621109 16736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:33:42.621417 16736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:42.878162 16591 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Refusing update from remote peer d0596cdb0a514cc08d7df0ceea528339: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260810 23:33:42.880277 16786 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260810 23:33:42.904984 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0596cdb0a514cc08d7df0ceea528339. Latest consensus state: current_term: 3 leader_uuid: "d0596cdb0a514cc08d7df0ceea528339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:42.905922 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:42.933431 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d0596cdb0a514cc08d7df0ceea528339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:42.934386 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:42.938879 16863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:42.939646 16863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:42.947688 16863 leader_election.cc:288] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:42.949903 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:42.951918 16786 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d0596cdb0a514cc08d7df0ceea528339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:42.952872 16786 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:42.955691 16351 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Refusing update from remote peer d0596cdb0a514cc08d7df0ceea528339: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260810 23:33:42.957988 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7" is_pre_election: true
I20260810 23:33:42.961237 16316 leader_election.cc:302] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61534c7c271c4e3393ff5b46a3498f9b; no voters: 0bfd786d28e24be390cdbbb4007290a7, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:42.963414 16861 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 23:33:42.996622 16863 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 23:33:43.002876 16874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0596cdb0a514cc08d7df0ceea528339. Latest consensus state: current_term: 3 leader_uuid: "d0596cdb0a514cc08d7df0ceea528339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:43.005133 16874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:43.008791 16881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d0596cdb0a514cc08d7df0ceea528339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:43.009704 16881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:43.148123 16736 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.129:39773
I20260810 23:33:43.148307 16976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.129:39773 every 8 connection(s)
I20260810 23:33:43.152663 16736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/data/info.pb
I20260810 23:33:43.155531 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 16736
I20260810 23:33:43.156374 12170 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/instance
I20260810 23:33:43.181599 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.226.130:0
--local_ip_for_outbound_sockets=127.11.226.130
--webserver_interface=127.11.226.130
--webserver_port=0
--tserver_master_addrs=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--builtin_ntp_servers=127.11.226.148:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260810 23:33:43.319012 16979 heartbeater.cc:347] Connected to a master server at 127.11.226.189:34939
I20260810 23:33:43.319645 16979 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:43.319824 16977 heartbeater.cc:347] Connected to a master server at 127.11.226.188:46109
I20260810 23:33:43.320413 16977 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:43.321641 16977 heartbeater.cc:511] Master 127.11.226.188:46109 requested a full tablet report, sending...
I20260810 23:33:43.323359 16982 heartbeater.cc:347] Connected to a master server at 127.11.226.190:42189
I20260810 23:33:43.323772 16982 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:43.324849 16982 heartbeater.cc:511] Master 127.11.226.190:42189 requested a full tablet report, sending...
I20260810 23:33:43.328218 16979 heartbeater.cc:511] Master 127.11.226.189:34939 requested a full tablet report, sending...
I20260810 23:33:43.329188 16334 ts_manager.cc:194] Registered new tserver with Master: c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773)
I20260810 23:33:43.327910 16578 ts_manager.cc:194] Registered new tserver with Master: c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773)
I20260810 23:33:43.331465 16451 ts_manager.cc:194] Registered new tserver with Master: c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773)
I20260810 23:33:43.336404 16451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.226.129:51079
I20260810 23:33:43.425239 16856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:33:43.425761 16856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:43.427614 16856 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
I20260810 23:33:43.429072 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:43.429733 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b" is_pre_election: true
I20260810 23:33:43.431991 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:43.432891 16856 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 23:33:43.528965 16861 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d0596cdb0a514cc08d7df0ceea528339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:43.534512 16861 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:43.843321 16979 heartbeater.cc:503] Master 127.11.226.189:34939 was elected leader, sending a full tablet report...
W20260810 23:33:43.906708 16991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:43.907482 16991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:43.908344 16991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:43.952731 16991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:43.953926 16991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.226.130
I20260810 23:33:44.000496 16991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.226.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.226.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.226.130
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:33:44.002607 16991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:44.005084 16991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:44.033064 17007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:44.035698 17012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:33:44.036494 17012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:44.038502 17012 leader_election.cc:288] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:44.039515 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:44.039849 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7" is_pre_election: true
I20260810 23:33:44.042651 16316 leader_election.cc:302] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61534c7c271c4e3393ff5b46a3498f9b; no voters: 0bfd786d28e24be390cdbbb4007290a7, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:44.043881 17012 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 23:33:44.486629 17019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:33:44.487165 17019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:44.489463 17019 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
I20260810 23:33:44.491024 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b" is_pre_election: true
I20260810 23:33:44.491366 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:44.493783 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:44.494993 17019 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 23:33:44.034904 17008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:44.037050 16991 server_base.cc:1034] running on GCE node
W20260810 23:33:44.035054 17010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:45.400069 16991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:45.403687 16991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:45.405256 16991 hybrid_clock.cc:648] HybridClock initialized: now 1786404825405208 us; error 90 us; skew 500 ppm
I20260810 23:33:45.406318 16991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:45.414593 16991 webserver.cc:460] Webserver started at http://127.11.226.130:36007/ using document root <none> and password file <none>
I20260810 23:33:45.415782 16991 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:45.416105 16991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:45.416800 16991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:45.423933 16991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/data/instance:
uuid: "80a651570d6e459d98bcf97054e07921"
format_stamp: "Formatted at 2026-08-10 23:33:45 on dist-test-slave-kvfs"
I20260810 23:33:45.425475 16991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/instance:
uuid: "80a651570d6e459d98bcf97054e07921"
format_stamp: "Formatted at 2026-08-10 23:33:45 on dist-test-slave-kvfs"
I20260810 23:33:45.437160 16991 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.009s sys 0.000s
I20260810 23:33:45.445324 17027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:45.446911 16991 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260810 23:33:45.447383 16991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal
uuid: "80a651570d6e459d98bcf97054e07921"
format_stamp: "Formatted at 2026-08-10 23:33:45 on dist-test-slave-kvfs"
I20260810 23:33:45.447829 16991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:45.497527 17032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:33:45.498049 17032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:45.500118 16991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:45.500959 17032 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
I20260810 23:33:45.501996 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b" is_pre_election: true
I20260810 23:33:45.502251 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:45.502620 16991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:45.503311 16991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:45.503990 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:45.504765 17032 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 23:33:45.506907 16991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:33:45.512435 16991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:33:45.512706 16991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:45.513031 16991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:33:45.513216 16991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:45.730470 16991 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.130:42517
I20260810 23:33:45.730563 17192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.130:42517 every 8 connection(s)
I20260810 23:33:45.734808 16991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/data/info.pb
I20260810 23:33:45.744302 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 16991
I20260810 23:33:45.745332 12170 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/instance
I20260810 23:33:45.785200 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.226.131:0
--local_ip_for_outbound_sockets=127.11.226.131
--webserver_interface=127.11.226.131
--webserver_port=0
--tserver_master_addrs=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--builtin_ntp_servers=127.11.226.148:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260810 23:33:45.836864 17193 heartbeater.cc:347] Connected to a master server at 127.11.226.188:46109
I20260810 23:33:45.837615 17193 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:45.839397 17193 heartbeater.cc:511] Master 127.11.226.188:46109 requested a full tablet report, sending...
I20260810 23:33:45.843783 16578 ts_manager.cc:194] Registered new tserver with Master: 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
I20260810 23:33:45.849208 17194 heartbeater.cc:347] Connected to a master server at 127.11.226.189:34939
I20260810 23:33:45.849632 17194 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:45.850620 17194 heartbeater.cc:511] Master 127.11.226.189:34939 requested a full tablet report, sending...
I20260810 23:33:45.853317 17195 heartbeater.cc:347] Connected to a master server at 127.11.226.190:42189
I20260810 23:33:45.853370 16451 ts_manager.cc:194] Registered new tserver with Master: 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
I20260810 23:33:45.853740 17195 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:45.854825 17195 heartbeater.cc:511] Master 127.11.226.190:42189 requested a full tablet report, sending...
I20260810 23:33:45.855823 16451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.226.130:39249
I20260810 23:33:45.858605 16334 ts_manager.cc:194] Registered new tserver with Master: 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
I20260810 23:33:46.029592 17205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:33:46.030157 17205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:46.032289 17205 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
I20260810 23:33:46.033891 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:46.033969 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b" is_pre_election: true
I20260810 23:33:46.036051 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:46.037029 17205 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 23:33:46.205662 17216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:33:46.206235 17216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:46.208858 17216 leader_election.cc:288] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:46.209613 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:46.211207 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7" is_pre_election: true
I20260810 23:33:46.213264 16316 leader_election.cc:302] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61534c7c271c4e3393ff5b46a3498f9b; no voters: 0bfd786d28e24be390cdbbb4007290a7, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:46.214172 17216 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 23:33:46.234294 17204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:46.235015 17204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:46.235766 17204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:46.277881 17204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:46.279125 17204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.226.131
I20260810 23:33:46.329123 17204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.226.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.226.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.226.131
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:33:46.330940 17204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:46.333514 17204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:33:46.361750 17194 heartbeater.cc:503] Master 127.11.226.189:34939 was elected leader, sending a full tablet report...
W20260810 23:33:46.364573 17221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:47.469038 17226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:33:47.469753 17226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:47.472075 17226 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
I20260810 23:33:47.473971 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b" is_pre_election: true
I20260810 23:33:47.474223 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:47.476197 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:47.476970 17226 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 23:33:47.578904 17228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:33:47.579504 17228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:47.581497 17228 leader_election.cc:288] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:47.583109 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7" is_pre_election: true
I20260810 23:33:47.583076 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:47.584887 16316 leader_election.cc:302] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61534c7c271c4e3393ff5b46a3498f9b; no voters: 0bfd786d28e24be390cdbbb4007290a7, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:47.585676 17228 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260810 23:33:46.366129 17220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:47.733727 17222 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20260810 23:33:46.368566 17223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:47.734935 17204 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 23:33:47.740491 17204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:47.743978 17204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:47.745718 17204 hybrid_clock.cc:648] HybridClock initialized: now 1786404827745642 us; error 103 us; skew 500 ppm
I20260810 23:33:47.746726 17204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:47.761826 17204 webserver.cc:460] Webserver started at http://127.11.226.131:43391/ using document root <none> and password file <none>
I20260810 23:33:47.763062 17204 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:47.763336 17204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:47.763926 17204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:47.770864 17204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/data/instance:
uuid: "2466b33eeb034688bcf670ff612d452a"
format_stamp: "Formatted at 2026-08-10 23:33:47 on dist-test-slave-kvfs"
I20260810 23:33:47.772655 17204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/instance:
uuid: "2466b33eeb034688bcf670ff612d452a"
format_stamp: "Formatted at 2026-08-10 23:33:47 on dist-test-slave-kvfs"
I20260810 23:33:47.782604 17204 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260810 23:33:47.790482 17233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:47.792192 17204 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:33:47.792722 17204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal
uuid: "2466b33eeb034688bcf670ff612d452a"
format_stamp: "Formatted at 2026-08-10 23:33:47 on dist-test-slave-kvfs"
I20260810 23:33:47.793244 17204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:47.846560 17204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:47.848829 17204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:47.849382 17204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:47.852741 17204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:33:47.858254 17204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:33:47.858505 17204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:47.858897 17204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:33:47.859138 17204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:48.053009 17204 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.131:39949
I20260810 23:33:48.053139 17405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.131:39949 every 8 connection(s)
I20260810 23:33:48.056602 17204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/data/info.pb
I20260810 23:33:48.067641 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 17204
I20260810 23:33:48.068434 12170 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/instance
I20260810 23:33:48.123235 17409 heartbeater.cc:347] Connected to a master server at 127.11.226.189:34939
I20260810 23:33:48.123842 17409 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:48.125336 17409 heartbeater.cc:511] Master 127.11.226.189:34939 requested a full tablet report, sending...
I20260810 23:33:48.126735 17410 heartbeater.cc:347] Connected to a master server at 127.11.226.190:42189
I20260810 23:33:48.127152 17410 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:48.127246 17406 heartbeater.cc:347] Connected to a master server at 127.11.226.188:46109
I20260810 23:33:48.127562 17406 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:48.128278 17410 heartbeater.cc:511] Master 127.11.226.190:42189 requested a full tablet report, sending...
I20260810 23:33:48.128470 17406 heartbeater.cc:511] Master 127.11.226.188:46109 requested a full tablet report, sending...
I20260810 23:33:48.129956 16451 ts_manager.cc:194] Registered new tserver with Master: 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:48.131551 16333 ts_manager.cc:194] Registered new tserver with Master: 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:48.131585 16574 ts_manager.cc:194] Registered new tserver with Master: 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:48.132493 16451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.226.131:43827
I20260810 23:33:48.146744 12170 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:33:48.203792 16451 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36352:
name: "default.Table_edb188d6a21846618b6fc3fe0927de7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:48.205221 16448 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36352:
name: "default.Table_bca6d3ee229b492798aa0ce50dd4c9f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:48.208658 16447 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36352:
name: "default.Table_edccecac3c7e4be4a5e3eab151d78907"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:48.210587 16446 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36352:
name: "default.Table_7c146274fcc84e8ea5b14582568f155f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:48.220449 16451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_edb188d6a21846618b6fc3fe0927de7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:48.228760 16448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:48.228772 16446 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7c146274fcc84e8ea5b14582568f155f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:48.229593 16447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_edccecac3c7e4be4a5e3eab151d78907 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:48.260962 12170 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu with pid 16391
W20260810 23:33:48.333390 17428 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.226.189:34939)
W20260810 23:33:48.408890 16979 heartbeater.cc:644] Failed to heartbeat to 127.11.226.189:34939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
W20260810 23:33:48.413795 17194 heartbeater.cc:644] Failed to heartbeat to 127.11.226.189:34939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:33:48.496804 17472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:33:48.497489 17472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:48.501401 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "61534c7c271c4e3393ff5b46a3498f9b" is_pre_election: true
I20260810 23:33:48.507599 17472 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:33:48.509341 16559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
W20260810 23:33:48.519709 16559 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:33:48.520459 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:48.521431 17472 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 23:33:48.595682 17476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:33:48.596314 17476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:48.599052 17476 leader_election.cc:288] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:48.600896 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7" is_pre_election: true
I20260810 23:33:48.601625 16591 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61534c7c271c4e3393ff5b46a3498f9b in term 3.
I20260810 23:33:48.606422 16316 leader_election.cc:302] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7, 61534c7c271c4e3393ff5b46a3498f9b; no voters:
I20260810 23:33:48.619565 17476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 23:33:48.620160 17476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Starting leader election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:33:48.620690 17476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 3 FOLLOWER]: Advancing to term 4
W20260810 23:33:48.622674 16314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20260810 23:33:48.629980 17476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
W20260810 23:33:48.635818 16314 leader_election.cc:334] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:33:48.637926 17409 heartbeater.cc:503] Master 127.11.226.189:34939 was elected leader, sending a full tablet report...
W20260810 23:33:48.653740 17409 heartbeater.cc:644] Failed to heartbeat to 127.11.226.189:34939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:33:48.662425 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61534c7c271c4e3393ff5b46a3498f9b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7"
I20260810 23:33:48.663127 16591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 3 FOLLOWER]: Advancing to term 4
I20260810 23:33:48.664199 17476 leader_election.cc:288] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 election: Requested vote from peers d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
W20260810 23:33:48.667867 16314 leader_election.cc:334] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:33:48.676071 16591 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61534c7c271c4e3393ff5b46a3498f9b in term 4.
I20260810 23:33:48.682837 16316 leader_election.cc:302] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7, 61534c7c271c4e3393ff5b46a3498f9b; no voters: d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:48.684468 17476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 4 FOLLOWER]: Leader election won for term 4
I20260810 23:33:48.702209 17476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [term 4 LEADER]: Becoming Leader. State: Replica: 61534c7c271c4e3393ff5b46a3498f9b, State: Running, Role: LEADER
I20260810 23:33:48.703481 17476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:48.718135 17481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61534c7c271c4e3393ff5b46a3498f9b. Latest consensus state: current_term: 4 leader_uuid: "61534c7c271c4e3393ff5b46a3498f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:48.719319 17481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:48.724841 17483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:33:48.797432 17483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:33:48.801774 17483 catalog_manager.cc:1259] Loaded cluster ID: 82c629e15cdf4d9cb19224cd28fe2c13
I20260810 23:33:48.802109 17483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:33:48.806133 17483 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:33:48.809408 17483 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Loaded TSK: 0
I20260810 23:33:48.810626 17483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:33:48.813642 17483 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260810 23:33:48.816817 17483 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260810 23:33:48.834559 16332 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33758:
name: "default.Table_bca6d3ee229b492798aa0ce50dd4c9f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:48.836288 16331 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33758:
name: "default.Table_edb188d6a21846618b6fc3fe0927de7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:48.835618 16333 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33758:
name: "default.Table_7c146274fcc84e8ea5b14582568f155f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:48.833791 16334 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33758:
name: "default.Table_edccecac3c7e4be4a5e3eab151d78907"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:48.878499 16331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_edb188d6a21846618b6fc3fe0927de7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:48.880545 16332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:48.895928 16333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7c146274fcc84e8ea5b14582568f155f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 23:33:48.900694 16334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_edccecac3c7e4be4a5e3eab151d78907 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:49.071885 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.189:34939
--webserver_interface=127.11.226.189
--webserver_port=39779
--builtin_ntp_servers=127.11.226.148:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--hive_metastore_uris=thrift://127.0.0.1:41643
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260810 23:33:49.081192 16332 catalog_manager.cc:2148] created HMS catalog entry for table [id=101fcdd599344d29bb0464af2be3c5cb]
I20260810 23:33:49.094877 16591 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 4 FOLLOWER]: Refusing update from remote peer 61534c7c271c4e3393ff5b46a3498f9b: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20260810 23:33:49.097396 17481 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260810 23:33:49.099072 16314 consensus_peers.cc:487] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b -> Peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Couldn't send request to peer d0596cdb0a514cc08d7df0ceea528339. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:49.133240 17489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61534c7c271c4e3393ff5b46a3498f9b. Latest consensus state: current_term: 4 leader_uuid: "61534c7c271c4e3393ff5b46a3498f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:49.134040 17489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:49.141386 17487 mvcc.cc:205] Tried to move back new op lower bound from 7317114179950419968 to 7317114178386219008. Current Snapshot: MvccSnapshot[applied={T|T < 7317114179950419968}]
I20260810 23:33:49.144176 16331 catalog_manager.cc:2148] created HMS catalog entry for table [id=54d3236abec942629222f30460ddaa97]
I20260810 23:33:49.153060 17489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "61534c7c271c4e3393ff5b46a3498f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:49.153856 17489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:49.170375 17476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "61534c7c271c4e3393ff5b46a3498f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:49.171252 17476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:49.190724 16332 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33758:
table { table_name: "default.Table_bca6d3ee229b492798aa0ce50dd4c9f8" } new_table_name: "default.Table_f348d0e4ead44bf8a9ad5adee8b016f3" modify_external_catalogs: true
I20260810 23:33:49.196698 16330 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33758:
table { table_name: "default.Table_edb188d6a21846618b6fc3fe0927de7c" } new_table_name: "default.Table_90d3ba7d3f7f4808accd7856a23d4348" modify_external_catalogs: true
I20260810 23:33:49.272609 17091 tablet_service.cc:1467] Processing CreateTablet for tablet 43dbcec7bf25426dac21a618f5f48028 (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.271790 17088 tablet_service.cc:1467] Processing CreateTablet for tablet 22d4f61906a14dc58a9c329b97c79d9a (DEFAULT_TABLE table=default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.281049 17091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43dbcec7bf25426dac21a618f5f48028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.283418 17088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d4f61906a14dc58a9c329b97c79d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.317195 16860 tablet_service.cc:1467] Processing CreateTablet for tablet 43dbcec7bf25426dac21a618f5f48028 (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.319769 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43dbcec7bf25426dac21a618f5f48028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.321435 16855 tablet_service.cc:1467] Processing CreateTablet for tablet 22d4f61906a14dc58a9c329b97c79d9a (DEFAULT_TABLE table=default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.323962 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d4f61906a14dc58a9c329b97c79d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.340498 17305 tablet_service.cc:1467] Processing CreateTablet for tablet 43dbcec7bf25426dac21a618f5f48028 (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.343999 17304 tablet_service.cc:1467] Processing CreateTablet for tablet 22d4f61906a14dc58a9c329b97c79d9a (DEFAULT_TABLE table=default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.345959 17304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d4f61906a14dc58a9c329b97c79d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.352339 17305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43dbcec7bf25426dac21a618f5f48028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.361003 17505 tablet_bootstrap.cc:492] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:33:49.375375 17505 tablet_bootstrap.cc:654] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:49.389863 16331 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33758:
name: "default.Table_585e701700f14afbb2df4af6ca5e03f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:49.390898 17505 log.cc:824] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:49.399627 16328 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33758:
name: "default.Table_0124b56cffb54cc0b4c79411d0526c06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:49.411548 17505 tablet_bootstrap.cc:492] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:33:49.412256 17505 ts_tablet_manager.cc:1397] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.052s user 0.022s sys 0.005s
I20260810 23:33:49.421851 17195 heartbeater.cc:503] Master 127.11.226.190:42189 was elected leader, sending a full tablet report...
I20260810 23:33:49.422317 16982 heartbeater.cc:503] Master 127.11.226.190:42189 was elected leader, sending a full tablet report...
I20260810 23:33:49.435760 17508 tablet_bootstrap.cc:492] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:33:49.439028 17515 tablet_bootstrap.cc:492] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:33:49.466698 17508 tablet_bootstrap.cc:654] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:49.475065 17508 log.cc:824] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:49.490475 16327 master_service.cc:946] ReplaceTablet: received request to replace tablet 43dbcec7bf25426dac21a618f5f48028 from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:49.498888 16326 master_service.cc:946] ReplaceTablet: received request to replace tablet 22d4f61906a14dc58a9c329b97c79d9a from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:49.512991 17515 tablet_bootstrap.cc:654] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:49.517248 17515 log.cc:824] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:49.528245 17508 tablet_bootstrap.cc:492] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:33:49.529284 17508 ts_tablet_manager.cc:1397] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.094s user 0.011s sys 0.012s
I20260810 23:33:49.533394 16327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Sending DeleteTablet for 3 replicas of tablet 43dbcec7bf25426dac21a618f5f48028
I20260810 23:33:49.537621 17515 tablet_bootstrap.cc:492] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:33:49.538005 16860 tablet_service.cc:1514] Processing DeleteTablet for tablet 43dbcec7bf25426dac21a618f5f48028 with delete_type TABLET_DATA_DELETED (replaced by tablet 14a2f1094d9d43aa8250f196faf19c69) from {username='slave'} at 127.0.0.1:52952
I20260810 23:33:49.538467 17515 ts_tablet_manager.cc:1397] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.100s user 0.011s sys 0.010s
I20260810 23:33:49.552762 17505 raft_consensus.cc:348] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:49.558836 16326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Sending DeleteTablet for 3 replicas of tablet 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:33:49.561554 16855 tablet_service.cc:1514] Processing DeleteTablet for tablet 22d4f61906a14dc58a9c329b97c79d9a with delete_type TABLET_DATA_DELETED (replaced by tablet 9ddfe4ea25ca44db94cbee975cce1334) from {username='slave'} at 127.0.0.1:52952
I20260810 23:33:49.562602 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet 22d4f61906a14dc58a9c329b97c79d9a with delete_type TABLET_DATA_DELETED (replaced by tablet 9ddfe4ea25ca44db94cbee975cce1334) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:49.563828 16326 catalog_manager.cc:6565] ReplaceTablet: tablet 22d4f61906a14dc58a9c329b97c79d9a deleted and replaced by tablet 9ddfe4ea25ca44db94cbee975cce1334
W20260810 23:33:49.579983 16314 catalog_manager.cc:4821] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): delete failed for tablet 22d4f61906a14dc58a9c329b97c79d9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22d4f61906a14dc58a9c329b97c79d9a already in progress: creating tablet
I20260810 23:33:49.562139 17091 tablet_service.cc:1514] Processing DeleteTablet for tablet 22d4f61906a14dc58a9c329b97c79d9a with delete_type TABLET_DATA_DELETED (replaced by tablet 9ddfe4ea25ca44db94cbee975cce1334) from {username='slave'} at 127.0.0.1:50894
I20260810 23:33:49.578126 17088 tablet_service.cc:1514] Processing DeleteTablet for tablet 43dbcec7bf25426dac21a618f5f48028 with delete_type TABLET_DATA_DELETED (replaced by tablet 14a2f1094d9d43aa8250f196faf19c69) from {username='slave'} at 127.0.0.1:50894
I20260810 23:33:49.587553 17505 raft_consensus.cc:374] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:49.588682 17505 raft_consensus.cc:729] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:33:49.589531 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet 43dbcec7bf25426dac21a618f5f48028 with delete_type TABLET_DATA_DELETED (replaced by tablet 14a2f1094d9d43aa8250f196faf19c69) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:49.590126 16327 catalog_manager.cc:6565] ReplaceTablet: tablet 43dbcec7bf25426dac21a618f5f48028 deleted and replaced by tablet 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:33:49.591847 16314 catalog_manager.cc:4821] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): delete failed for tablet 43dbcec7bf25426dac21a618f5f48028 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43dbcec7bf25426dac21a618f5f48028 already in progress: creating tablet
I20260810 23:33:49.593338 17505 consensus_queue.cc:260] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
W20260810 23:33:49.601943 16311 catalog_manager.cc:4821] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): delete failed for tablet 22d4f61906a14dc58a9c329b97c79d9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22d4f61906a14dc58a9c329b97c79d9a already in progress: creating tablet
W20260810 23:33:49.604315 16311 catalog_manager.cc:4821] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): delete failed for tablet 43dbcec7bf25426dac21a618f5f48028 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43dbcec7bf25426dac21a618f5f48028 already in progress: creating tablet
I20260810 23:33:49.615238 16855 tablet_service.cc:1467] Processing CreateTablet for tablet 14a2f1094d9d43aa8250f196faf19c69 (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.616442 17505 ts_tablet_manager.cc:1428] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.204s user 0.039s sys 0.010s
I20260810 23:33:49.616971 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a2f1094d9d43aa8250f196faf19c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.624866 17505 tablet_bootstrap.cc:492] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:33:49.635478 17505 tablet_bootstrap.cc:654] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:49.638826 17304 tablet_service.cc:1467] Processing CreateTablet for tablet 14a2f1094d9d43aa8250f196faf19c69 (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.640333 17304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a2f1094d9d43aa8250f196faf19c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.647172 17091 tablet_service.cc:1514] Processing DeleteTablet for tablet 43dbcec7bf25426dac21a618f5f48028 with delete_type TABLET_DATA_DELETED (replaced by tablet 14a2f1094d9d43aa8250f196faf19c69) from {username='slave'} at 127.0.0.1:50894
I20260810 23:33:49.671754 17522 tablet_replica.cc:330] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:33:49.672827 17522 raft_consensus.cc:2229] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:49.673681 17522 raft_consensus.cc:2258] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:49.648794 17088 tablet_service.cc:1467] Processing CreateTablet for tablet 14a2f1094d9d43aa8250f196faf19c69 (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.676211 17088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a2f1094d9d43aa8250f196faf19c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.686866 17522 ts_tablet_manager.cc:1905] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:33:49.690050 16316 catalog_manager.cc:4821] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): delete failed for tablet 43dbcec7bf25426dac21a618f5f48028 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43dbcec7bf25426dac21a618f5f48028 already in progress: creating tablet
W20260810 23:33:49.692657 16316 catalog_manager.cc:4821] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): delete failed for tablet 22d4f61906a14dc58a9c329b97c79d9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22d4f61906a14dc58a9c329b97c79d9a already in progress: creating tablet
I20260810 23:33:49.702834 17410 heartbeater.cc:503] Master 127.11.226.190:42189 was elected leader, sending a full tablet report...
I20260810 23:33:49.711417 17515 raft_consensus.cc:348] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:49.712914 17515 raft_consensus.cc:374] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:49.713279 17515 raft_consensus.cc:729] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:33:49.714407 17515 consensus_queue.cc:260] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
W20260810 23:33:49.736263 16314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:34939, user_credentials={real_user=slave}} blocked reactor thread for 55516.7us
I20260810 23:33:49.738595 17305 tablet_service.cc:1514] Processing DeleteTablet for tablet 43dbcec7bf25426dac21a618f5f48028 with delete_type TABLET_DATA_DELETED (replaced by tablet 14a2f1094d9d43aa8250f196faf19c69) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:49.739279 17303 tablet_service.cc:1514] Processing DeleteTablet for tablet 22d4f61906a14dc58a9c329b97c79d9a with delete_type TABLET_DATA_DELETED (replaced by tablet 9ddfe4ea25ca44db94cbee975cce1334) from {username='slave'} at 127.0.0.1:55552
W20260810 23:33:49.741214 16314 catalog_manager.cc:4821] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): delete failed for tablet 43dbcec7bf25426dac21a618f5f48028 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43dbcec7bf25426dac21a618f5f48028 already in progress: creating tablet
W20260810 23:33:49.742233 16314 catalog_manager.cc:4821] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): delete failed for tablet 22d4f61906a14dc58a9c329b97c79d9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22d4f61906a14dc58a9c329b97c79d9a already in progress: creating tablet
I20260810 23:33:49.747088 17505 tablet_bootstrap.cc:492] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:33:49.747656 17505 ts_tablet_manager.cc:1397] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.123s user 0.012s sys 0.001s
I20260810 23:33:49.750748 17505 raft_consensus.cc:348] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:49.751590 17505 raft_consensus.cc:374] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:49.754765 17091 tablet_service.cc:1467] Processing CreateTablet for tablet 9ddfe4ea25ca44db94cbee975cce1334 (DEFAULT_TABLE table=default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.756290 17091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ddfe4ea25ca44db94cbee975cce1334. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.760479 16860 tablet_service.cc:1467] Processing CreateTablet for tablet 9ddfe4ea25ca44db94cbee975cce1334 (DEFAULT_TABLE table=default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.751904 17505 raft_consensus.cc:729] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:33:49.766477 17515 ts_tablet_manager.cc:1428] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.228s user 0.035s sys 0.020s
I20260810 23:33:49.766891 17505 consensus_queue.cc:260] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:49.767974 17515 tablet_bootstrap.cc:492] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:33:49.768518 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ddfe4ea25ca44db94cbee975cce1334. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.769136 17505 ts_tablet_manager.cc:1428] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.021s user 0.005s sys 0.003s
W20260810 23:33:49.784534 16985 tablet.cc:2369] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:33:49.791018 16326 master_service.cc:946] ReplaceTablet: received request to replace tablet 14a2f1094d9d43aa8250f196faf19c69 from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:49.794562 17515 tablet_bootstrap.cc:654] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:49.803803 16325 master_service.cc:946] ReplaceTablet: received request to replace tablet 14a2f1094d9d43aa8250f196faf19c69 from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:49.814684 16855 tablet_service.cc:1514] Processing DeleteTablet for tablet 43dbcec7bf25426dac21a618f5f48028 with delete_type TABLET_DATA_DELETED (replaced by tablet 14a2f1094d9d43aa8250f196faf19c69) from {username='slave'} at 127.0.0.1:52952
I20260810 23:33:49.820583 17087 tablet_service.cc:1514] Processing DeleteTablet for tablet 22d4f61906a14dc58a9c329b97c79d9a with delete_type TABLET_DATA_DELETED (replaced by tablet 9ddfe4ea25ca44db94cbee975cce1334) from {username='slave'} at 127.0.0.1:50894
I20260810 23:33:49.821280 16326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Sending DeleteTablet for 3 replicas of tablet 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:33:49.822930 16326 catalog_manager.cc:6565] ReplaceTablet: tablet 14a2f1094d9d43aa8250f196faf19c69 deleted and replaced by tablet a72d0f3db05b4aabb69a46a8bc88f61e
I20260810 23:33:49.825143 16855 tablet_service.cc:1514] Processing DeleteTablet for tablet 14a2f1094d9d43aa8250f196faf19c69 with delete_type TABLET_DATA_DELETED (replaced by tablet a72d0f3db05b4aabb69a46a8bc88f61e) from {username='slave'} at 127.0.0.1:52952
I20260810 23:33:49.826674 17087 tablet_service.cc:1514] Processing DeleteTablet for tablet 14a2f1094d9d43aa8250f196faf19c69 with delete_type TABLET_DATA_DELETED (replaced by tablet a72d0f3db05b4aabb69a46a8bc88f61e) from {username='slave'} at 127.0.0.1:50894
I20260810 23:33:49.829659 17303 tablet_service.cc:1514] Processing DeleteTablet for tablet 14a2f1094d9d43aa8250f196faf19c69 with delete_type TABLET_DATA_DELETED (replaced by tablet a72d0f3db05b4aabb69a46a8bc88f61e) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:49.830590 17515 tablet_bootstrap.cc:492] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:33:49.831126 17515 ts_tablet_manager.cc:1397] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.003s
I20260810 23:33:49.834074 17515 raft_consensus.cc:348] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:49.834270 17303 tablet_service.cc:1467] Processing CreateTablet for tablet 9ddfe4ea25ca44db94cbee975cce1334 (DEFAULT_TABLE table=default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.834824 17515 raft_consensus.cc:374] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:49.835145 17515 raft_consensus.cc:729] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:33:49.835706 17303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ddfe4ea25ca44db94cbee975cce1334. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:49.840591 16314 catalog_manager.cc:4821] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): delete failed for tablet 14a2f1094d9d43aa8250f196faf19c69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14a2f1094d9d43aa8250f196faf19c69 already in progress: creating tablet
I20260810 23:33:49.835899 17515 consensus_queue.cc:260] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:49.846594 17515 ts_tablet_manager.cc:1428] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
W20260810 23:33:49.853384 17200 tablet.cc:2369] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:33:49.857774 16855 tablet_service.cc:1514] Processing DeleteTablet for tablet 22d4f61906a14dc58a9c329b97c79d9a with delete_type TABLET_DATA_DELETED (replaced by tablet 9ddfe4ea25ca44db94cbee975cce1334) from {username='slave'} at 127.0.0.1:52952
I20260810 23:33:49.860733 17520 tablet_replica.cc:330] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:33:49.861774 17520 raft_consensus.cc:2229] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:49.862442 17520 raft_consensus.cc:2258] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:49.863905 17515 tablet_bootstrap.cc:492] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:33:49.757951 17508 raft_consensus.cc:348] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:49.868854 17508 raft_consensus.cc:374] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:49.869357 17508 raft_consensus.cc:729] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:49.870769 17508 consensus_queue.cc:260] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:49.874030 17515 tablet_bootstrap.cc:654] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:49.876426 17520 ts_tablet_manager.cc:1905] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:49.881526 17505 tablet_bootstrap.cc:492] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:33:49.888584 17505 tablet_bootstrap.cc:654] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
23:33:49.888 [HMS - ERROR - pool-6-thread-3] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_7c146274fcc84e8ea5b14582568f155f already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
I20260810 23:33:49.910907 17508 ts_tablet_manager.cc:1428] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.381s user 0.036s sys 0.008s
I20260810 23:33:49.911192 17515 tablet_bootstrap.cc:492] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:33:49.911918 17515 ts_tablet_manager.cc:1397] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.007s
W20260810 23:33:49.912179 16333 catalog_manager.cc:2144] Already present: failed to create HMS catalog entry for table [id=b1f05dfcf93d4748ac0673d4e0a77943]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_7c146274fcc84e8ea5b14582568f155f already exists)
I20260810 23:33:49.913363 16333 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33758 (request call id 22) took 1081 ms. Trace:
I20260810 23:33:49.913812 16333 rpcz_store.cc:276] 0810 23:33:48.831967 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:48.832167 (+ 200us) service_pool.cc:227] Handling call
0810 23:33:48.896560 (+ 64393us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:33:48.898596 (+ 2036us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:33:49.913255 (+1014659us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":244896,"Hive Metastore.run_cpu_time_us":2606,"Hive Metastore.run_wall_time_us":767032}
I20260810 23:33:49.920150 17508 tablet_bootstrap.cc:492] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:33:49.915544 17515 raft_consensus.cc:348] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:33:49.922995 17515 raft_consensus.cc:374] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:49.923309 17515 raft_consensus.cc:729] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:33:49.924062 17515 consensus_queue.cc:260] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:33:49.929417 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet 43dbcec7bf25426dac21a618f5f48028 with delete_type TABLET_DATA_DELETED (replaced by tablet 14a2f1094d9d43aa8250f196faf19c69) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:49.930539 17521 tablet_replica.cc:330] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:33:49.931522 17521 raft_consensus.cc:2229] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:49.931622 17520 ts_tablet_manager.cc:1918] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:49.932199 17520 log.cc:1196] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/43dbcec7bf25426dac21a618f5f48028
I20260810 23:33:49.932252 17521 raft_consensus.cc:2258] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:49.933280 17520 ts_tablet_manager.cc:1939] T 43dbcec7bf25426dac21a618f5f48028 P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:33:49.937150 17505 tablet_bootstrap.cc:492] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:33:49.937207 17515 ts_tablet_manager.cc:1428] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260810 23:33:49.937697 17505 ts_tablet_manager.cc:1397] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.056s user 0.015s sys 0.008s
I20260810 23:33:49.938263 17521 ts_tablet_manager.cc:1905] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:49.941099 17505 raft_consensus.cc:348] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:33:49.942137 17505 raft_consensus.cc:374] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:49.942436 17505 raft_consensus.cc:729] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:33:49.943145 17505 consensus_queue.cc:260] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:33:49.946686 17505 ts_tablet_manager.cc:1428] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 23:33:49.947758 17505 tablet_bootstrap.cc:492] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:33:49.951126 16316 catalog_manager.cc:4841] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): tablet 43dbcec7bf25426dac21a618f5f48028 (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
I20260810 23:33:49.955804 17520 tablet_replica.cc:330] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:33:49.959028 17520 raft_consensus.cc:2229] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:49.962226 17520 raft_consensus.cc:2258] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:49.967465 17505 tablet_bootstrap.cc:654] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:49.969002 17520 ts_tablet_manager.cc:1905] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:49.971616 16333 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33758:
name: "default.Table_bf8e0e4f072044999c5d464208e52921"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:49.980369 17508 tablet_bootstrap.cc:654] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:49.987876 17088 tablet_service.cc:1467] Processing CreateTablet for tablet a72d0f3db05b4aabb69a46a8bc88f61e (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.988130 16855 tablet_service.cc:1467] Processing CreateTablet for tablet a72d0f3db05b4aabb69a46a8bc88f61e (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.989235 17088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72d0f3db05b4aabb69a46a8bc88f61e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.989410 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72d0f3db05b4aabb69a46a8bc88f61e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.990159 17505 tablet_bootstrap.cc:492] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:33:49.990628 17505 ts_tablet_manager.cc:1397] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.007s
I20260810 23:33:49.993777 17505 raft_consensus.cc:348] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:49.994020 17303 tablet_service.cc:1467] Processing CreateTablet for tablet a72d0f3db05b4aabb69a46a8bc88f61e (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:49.994547 17505 raft_consensus.cc:374] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:49.994831 17505 raft_consensus.cc:729] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:33:49.995256 17303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72d0f3db05b4aabb69a46a8bc88f61e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:49.995507 17505 consensus_queue.cc:260] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.001783 17505 ts_tablet_manager.cc:1428] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260810 23:33:50.014549 17522 ts_tablet_manager.cc:1918] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:50.015232 17522 log.cc:1196] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/43dbcec7bf25426dac21a618f5f48028
I20260810 23:33:50.016508 17522 ts_tablet_manager.cc:1939] T 43dbcec7bf25426dac21a618f5f48028 P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:33:50.040370 17515 tablet_bootstrap.cc:492] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:33:50.042158 16327 master_service.cc:946] ReplaceTablet: received request to replace tablet 14a2f1094d9d43aa8250f196faf19c69 from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:50.043107 16326 master_service.cc:946] ReplaceTablet: received request to replace tablet 22d4f61906a14dc58a9c329b97c79d9a from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:50.049403 17515 tablet_bootstrap.cc:654] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:50.057175 17508 tablet_bootstrap.cc:492] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:33:50.057714 17508 ts_tablet_manager.cc:1397] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.138s user 0.007s sys 0.005s
I20260810 23:33:50.061992 16311 catalog_manager.cc:4841] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): tablet 43dbcec7bf25426dac21a618f5f48028 (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
I20260810 23:33:50.071486 17522 tablet_replica.cc:330] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:33:50.072626 17522 raft_consensus.cc:2229] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:50.073194 17522 raft_consensus.cc:2258] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:50.081835 17520 ts_tablet_manager.cc:1918] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:50.082302 17520 log.cc:1196] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/14a2f1094d9d43aa8250f196faf19c69
I20260810 23:33:50.083004 17520 ts_tablet_manager.cc:1939] T 14a2f1094d9d43aa8250f196faf19c69 P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:33:50.087214 17515 tablet_bootstrap.cc:492] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:33:50.087725 17515 ts_tablet_manager.cc:1397] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.005s
I20260810 23:33:50.093061 17522 ts_tablet_manager.cc:1905] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:50.077252 17508 raft_consensus.cc:348] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.096292 17508 raft_consensus.cc:374] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:50.096622 17508 raft_consensus.cc:729] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:50.100593 17521 ts_tablet_manager.cc:1918] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:50.101154 17521 log.cc:1196] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/43dbcec7bf25426dac21a618f5f48028
I20260810 23:33:50.102252 17521 ts_tablet_manager.cc:1939] T 43dbcec7bf25426dac21a618f5f48028 P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:33:50.102221 16316 catalog_manager.cc:4841] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): tablet 14a2f1094d9d43aa8250f196faf19c69 (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
I20260810 23:33:50.104149 17515 raft_consensus.cc:348] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.104956 17515 raft_consensus.cc:374] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:50.105237 17515 raft_consensus.cc:729] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:33:50.104386 17508 consensus_queue.cc:260] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.107893 17508 ts_tablet_manager.cc:1428] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.050s user 0.002s sys 0.006s
I20260810 23:33:50.108870 17508 tablet_bootstrap.cc:492] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
W20260810 23:33:50.111712 16326 catalog_manager.cc:5328] Ignoring report from unknown tablet 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:33:50.119267 17505 tablet_bootstrap.cc:492] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:33:50.119267 17520 tablet_replica.cc:330] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:33:50.105937 17515 consensus_queue.cc:260] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.126381 17505 tablet_bootstrap.cc:654] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:50.128115 17515 ts_tablet_manager.cc:1428] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.040s user 0.009s sys 0.000s
I20260810 23:33:50.129244 17515 tablet_bootstrap.cc:492] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
W20260810 23:33:50.126433 16326 catalog_manager.cc:5328] Ignoring report from unknown tablet 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:33:50.120131 17520 raft_consensus.cc:2229] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:50.132499 17520 raft_consensus.cc:2258] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260810 23:33:50.132429 17488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:50.133337 17488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:50.134057 17488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:33:50.143752 17505 tablet_bootstrap.cc:492] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:33:50.144483 17505 ts_tablet_manager.cc:1397] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.004s
I20260810 23:33:50.147862 17505 raft_consensus.cc:348] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.148736 17505 raft_consensus.cc:374] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:50.149024 17505 raft_consensus.cc:729] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:33:50.149765 17505 consensus_queue.cc:260] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.157563 17505 ts_tablet_manager.cc:1428] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260810 23:33:50.170766 17520 ts_tablet_manager.cc:1905] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:50.174904 17524 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:50.175596 17524 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:33:50.187137 17524 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:50.212911 17515 tablet_bootstrap.cc:654] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:50.219518 16314 catalog_manager.cc:4841] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): tablet 43dbcec7bf25426dac21a618f5f48028 (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
I20260810 23:33:50.219722 17508 tablet_bootstrap.cc:654] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:50.244215 17522 ts_tablet_manager.cc:1918] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:50.245275 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a2f1094d9d43aa8250f196faf19c69" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2466b33eeb034688bcf670ff612d452a" is_pre_election: true
W20260810 23:33:50.247788 17030 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:50.244786 17522 log.cc:1196] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:33:50.254472 17515 tablet_bootstrap.cc:492] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:33:50.254819 17522 ts_tablet_manager.cc:1939] T 22d4f61906a14dc58a9c329b97c79d9a P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:33:50.255069 17515 ts_tablet_manager.cc:1397] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.126s user 0.012s sys 0.002s
I20260810 23:33:50.257910 17515 raft_consensus.cc:348] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.258692 17515 raft_consensus.cc:374] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:50.264557 17522 tablet_replica.cc:330] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:33:50.265484 16311 catalog_manager.cc:4841] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): tablet 22d4f61906a14dc58a9c329b97c79d9a (table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]) successfully deleted
W20260810 23:33:50.270640 17031 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:33:50.271340 17031 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a651570d6e459d98bcf97054e07921; no voters: 2466b33eeb034688bcf670ff612d452a, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:33:50.272933 17524 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:33:50.275285 17488 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260810 23:33:50.275655 17488 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
I20260810 23:33:50.275573 17522 raft_consensus.cc:2229] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260810 23:33:50.275964 17488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20260810 23:33:50.276212 17522 raft_consensus.cc:2258] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260810 23:33:50.276373 17488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:50.276759 17488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:33:50.277109 17488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:33:50.279428 17522 ts_tablet_manager.cc:1905] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:50.283525 17508 tablet_bootstrap.cc:492] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:33:50.284415 17508 ts_tablet_manager.cc:1397] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.176s user 0.015s sys 0.007s
I20260810 23:33:50.287097 17508 raft_consensus.cc:348] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:33:50.258960 17515 raft_consensus.cc:729] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:33:50.290026 17515 consensus_queue.cc:260] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.298784 17520 ts_tablet_manager.cc:1918] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:50.299265 17520 log.cc:1196] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:33:50.302253 16326 master_service.cc:946] ReplaceTablet: received request to replace tablet 9ddfe4ea25ca44db94cbee975cce1334 from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:50.287876 17508 raft_consensus.cc:374] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:50.304804 17508 raft_consensus.cc:729] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:50.305840 17508 consensus_queue.cc:260] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:33:50.318972 16327 master_service.cc:946] ReplaceTablet: received request to replace tablet 22d4f61906a14dc58a9c329b97c79d9a from {username='slave'} at 127.0.0.1:33758
W20260810 23:33:50.323663 16327 catalog_manager.cc:5328] Ignoring report from unknown tablet 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:33:50.328620 17508 ts_tablet_manager.cc:1428] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260810 23:33:50.333292 16326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Sending DeleteTablet for 3 replicas of tablet 9ddfe4ea25ca44db94cbee975cce1334
I20260810 23:33:50.299995 17520 ts_tablet_manager.cc:1939] T 22d4f61906a14dc58a9c329b97c79d9a P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:33:50.338927 16316 catalog_manager.cc:4841] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): tablet 22d4f61906a14dc58a9c329b97c79d9a (table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]) successfully deleted
I20260810 23:33:50.335813 16860 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ddfe4ea25ca44db94cbee975cce1334 with delete_type TABLET_DATA_DELETED (replaced by tablet d31ab1862b084678b90352f9c8e242e3) from {username='slave'} at 127.0.0.1:52952
I20260810 23:33:50.302649 17515 ts_tablet_manager.cc:1428] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.047s user 0.005s sys 0.001s
W20260810 23:33:50.342494 17420 tablet.cc:2369] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:33:50.348699 17088 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ddfe4ea25ca44db94cbee975cce1334 with delete_type TABLET_DATA_DELETED (replaced by tablet d31ab1862b084678b90352f9c8e242e3) from {username='slave'} at 127.0.0.1:50894
I20260810 23:33:50.358150 17508 tablet_bootstrap.cc:492] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
W20260810 23:33:50.358943 16325 catalog_manager.cc:5328] Ignoring report from unknown tablet 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:33:50.360841 16326 catalog_manager.cc:6565] ReplaceTablet: tablet 9ddfe4ea25ca44db94cbee975cce1334 deleted and replaced by tablet d31ab1862b084678b90352f9c8e242e3
I20260810 23:33:50.362154 17488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41643
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.189:34939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.226.189
--webserver_port=39779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:33:50.362646 17303 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ddfe4ea25ca44db94cbee975cce1334 with delete_type TABLET_DATA_DELETED (replaced by tablet d31ab1862b084678b90352f9c8e242e3) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:50.360229 17520 tablet_replica.cc:330] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:33:50.364604 17488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:50.365124 17520 raft_consensus.cc:2229] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:50.365964 17520 raft_consensus.cc:2258] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:50.368299 17488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:50.369084 16314 catalog_manager.cc:4821] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): delete failed for tablet 9ddfe4ea25ca44db94cbee975cce1334 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ddfe4ea25ca44db94cbee975cce1334 already in progress: creating tablet
I20260810 23:33:50.373294 17520 ts_tablet_manager.cc:1905] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:50.377105 17508 tablet_bootstrap.cc:654] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:50.395435 16860 tablet_service.cc:1467] Processing CreateTablet for tablet d31ab1862b084678b90352f9c8e242e3 (DEFAULT_TABLE table=default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:50.396497 17088 tablet_service.cc:1467] Processing CreateTablet for tablet d31ab1862b084678b90352f9c8e242e3 (DEFAULT_TABLE table=default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:50.397029 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31ab1862b084678b90352f9c8e242e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:50.397786 17088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31ab1862b084678b90352f9c8e242e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:50.407704 17303 tablet_service.cc:1467] Processing CreateTablet for tablet d31ab1862b084678b90352f9c8e242e3 (DEFAULT_TABLE table=default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:50.409997 17522 ts_tablet_manager.cc:1918] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:50.410650 17522 log.cc:1196] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/14a2f1094d9d43aa8250f196faf19c69
I20260810 23:33:50.410693 17303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31ab1862b084678b90352f9c8e242e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:50.414425 17522 ts_tablet_manager.cc:1939] T 14a2f1094d9d43aa8250f196faf19c69 P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:33:50.418268 16311 catalog_manager.cc:4841] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): tablet 14a2f1094d9d43aa8250f196faf19c69 (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
I20260810 23:33:50.419628 17522 tablet_replica.cc:330] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:33:50.420929 17522 raft_consensus.cc:2229] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:50.421577 17522 raft_consensus.cc:2258] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:50.427598 17522 ts_tablet_manager.cc:1905] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:33:50.430737 17565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:50.435070 17508 tablet_bootstrap.cc:492] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:33:50.435743 17508 ts_tablet_manager.cc:1397] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.078s user 0.014s sys 0.000s
I20260810 23:33:50.453473 17522 ts_tablet_manager.cc:1918] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:50.454134 17522 log.cc:1196] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/9ddfe4ea25ca44db94cbee975cce1334
I20260810 23:33:50.455045 17522 ts_tablet_manager.cc:1939] T 9ddfe4ea25ca44db94cbee975cce1334 P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:33:50.439258 17508 raft_consensus.cc:348] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.455677 17508 raft_consensus.cc:374] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:50.456144 17508 raft_consensus.cc:729] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:50.458673 16311 catalog_manager.cc:4841] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): tablet 9ddfe4ea25ca44db94cbee975cce1334 (table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]) successfully deleted
I20260810 23:33:50.460242 17508 consensus_queue.cc:260] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.469319 17505 tablet_bootstrap.cc:492] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:33:50.470258 16334 catalog_manager.cc:2148] created HMS catalog entry for table [id=2a70ac3899e649deb7fbd6e3bbf1d43d]
I20260810 23:33:50.478574 17505 tablet_bootstrap.cc:654] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:50.483438 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ddfe4ea25ca44db94cbee975cce1334 with delete_type TABLET_DATA_DELETED (replaced by tablet d31ab1862b084678b90352f9c8e242e3) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:50.483752 17524 raft_consensus.cc:482] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:50.484349 17524 raft_consensus.cc:504] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
W20260810 23:33:50.485847 16314 catalog_manager.cc:4821] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): delete failed for tablet 9ddfe4ea25ca44db94cbee975cce1334 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ddfe4ea25ca44db94cbee975cce1334 already in progress: creating tablet
I20260810 23:33:50.482589 17508 ts_tablet_manager.cc:1428] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.046s user 0.006s sys 0.002s
I20260810 23:33:50.488706 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72d0f3db05b4aabb69a46a8bc88f61e" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2466b33eeb034688bcf670ff612d452a" is_pre_election: true
I20260810 23:33:50.489637 17515 tablet_bootstrap.cc:492] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:33:50.489682 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72d0f3db05b4aabb69a46a8bc88f61e" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36" is_pre_election: true
W20260810 23:33:50.490631 17030 leader_election.cc:341] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:50.491031 16904 raft_consensus.cc:2454] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a651570d6e459d98bcf97054e07921 in term 0.
I20260810 23:33:50.491312 17524 leader_election.cc:288] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:50.492594 17031 leader_election.cc:302] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36; no voters: 2466b33eeb034688bcf670ff612d452a
I20260810 23:33:50.498560 17515 tablet_bootstrap.cc:654] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:50.500556 17505 tablet_bootstrap.cc:492] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:33:50.501138 17505 ts_tablet_manager.cc:1397] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.008s
I20260810 23:33:50.502388 16334 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33758 (request call id 23) took 1669 ms. Trace:
I20260810 23:33:50.503751 17524 raft_consensus.cc:2781] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:50.504216 17524 raft_consensus.cc:482] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:50.504619 17524 raft_consensus.cc:3037] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:50.502825 16334 rpcz_store.cc:276] 0810 23:33:48.832940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0810 23:33:48.833086 (+ 146us) service_pool.cc:227] Handling call
0810 23:33:48.901204 (+ 68118us) catalog_manager.cc:2075] Acquired catalog manager lock
0810 23:33:48.903160 (+ 1956us) catalog_manager.cc:2121] Created new table and tablet info
0810 23:33:50.470202 (+1567042us) catalog_manager.cc:2147] Created new table in HMS catalog
0810 23:33:50.501387 (+ 31185us) catalog_manager.cc:2181] Wrote table and tablets to system table
0810 23:33:50.501878 (+ 491us) catalog_manager.cc:2215] Inserted table and tablets into CatalogManager maps
0810 23:33:50.502297 (+ 419us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0810 23:33:50.472573 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0810 23:33:50.473103 (+ 530us) write_op.cc:432] Acquiring schema lock in shared mode
0810 23:33:50.473133 (+ 30us) write_op.cc:435] Acquired schema lock
0810 23:33:50.473150 (+ 17us) tablet.cc:660] Decoding operations
0810 23:33:50.473521 (+ 371us) write_op.cc:620] Acquiring the partition lock for write op
0810 23:33:50.473572 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0810 23:33:50.473599 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0810 23:33:50.473856 (+ 257us) tablet.cc:699] Row locks acquired
0810 23:33:50.473880 (+ 24us) write_op.cc:260] PREPARE: finished
0810 23:33:50.474070 (+ 190us) write_op.cc:270] Start()
0810 23:33:50.474184 (+ 114us) write_op.cc:276] Timestamp: P: 1786404830474016 usec, L: 0
0810 23:33:50.474204 (+ 20us) op_driver.cc:331] REPLICATION: starting
0810 23:33:50.474892 (+ 688us) log.cc:842] Serialized 432 byte log entry
0810 23:33:50.499017 (+ 24125us) op_driver.cc:444] REPLICATION: finished
0810 23:33:50.499534 (+ 517us) write_op.cc:301] APPLY: starting
0810 23:33:50.499609 (+ 75us) tablet.cc:1365] starting BulkCheckPresence
0810 23:33:50.499737 (+ 128us) tablet.cc:1368] finished BulkCheckPresence
0810 23:33:50.499852 (+ 115us) tablet.cc:1370] starting ApplyRowOperation cycle
0810 23:33:50.500348 (+ 496us) tablet.cc:1381] finished ApplyRowOperation cycle
0810 23:33:50.500511 (+ 163us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0810 23:33:50.500557 (+ 46us) write_op.cc:312] APPLY: finished
0810 23:33:50.500796 (+ 239us) log.cc:842] Serialized 28 byte log entry
0810 23:33:50.501001 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:50.501091 (+ 90us) write_op.cc:454] Released schema lock
0810 23:33:50.501134 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0810 23:33:50.501356 (+ 222us) write_op.cc:489] Releasing partition, row and schema locks
0810 23:33:50.501383 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1007502,"Hive Metastore.run_cpu_time_us":771,"Hive Metastore.run_wall_time_us":558479,"child_traces":[["op",{"apply.queue_time_us":313,"apply.run_cpu_time_us":2048,"apply.run_wall_time_us":2047,"num_ops":2,"prepare.queue_time_us":134,"prepare.run_cpu_time_us":2771,"prepare.run_wall_time_us":2771,"raft.queue_time_us":5381,"raft.run_cpu_time_us":906,"raft.run_wall_time_us":908,"replication_time_us":24707}]]}
I20260810 23:33:50.510888 17505 raft_consensus.cc:348] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.511873 17505 raft_consensus.cc:374] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:50.516973 17505 raft_consensus.cc:729] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:33:50.517602 17534 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:50.518507 17534 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.521580 17505 consensus_queue.cc:260] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.529253 16327 master_service.cc:946] ReplaceTablet: received request to replace tablet d31ab1862b084678b90352f9c8e242e3 from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:50.531486 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ddfe4ea25ca44db94cbee975cce1334 with delete_type TABLET_DATA_DELETED (replaced by tablet d31ab1862b084678b90352f9c8e242e3) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:50.531674 16334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33758:
table { table_name: "default.Table_edccecac3c7e4be4a5e3eab151d78907" } modify_external_catalogs: true
I20260810 23:33:50.532616 17521 tablet_replica.cc:330] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:33:50.532631 16334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33758:
table { table_name: "default.Table_edccecac3c7e4be4a5e3eab151d78907" } modify_external_catalogs: true
I20260810 23:33:50.533505 17521 raft_consensus.cc:2229] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:50.534065 17521 raft_consensus.cc:2258] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:50.536520 17521 ts_tablet_manager.cc:1905] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:50.561664 17534 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
I20260810 23:33:50.569706 16325 master_service.cc:946] ReplaceTablet: received request to replace tablet 9ddfe4ea25ca44db94cbee975cce1334 from {username='slave'} at 127.0.0.1:33758
F20260810 23:33:50.574621 16731 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_bca6d3ee229b492798aa0ce50dd4c9f8
*** Check failure stack trace: ***
I20260810 23:33:50.581768 17520 ts_tablet_manager.cc:1918] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:50.590348 17524 raft_consensus.cc:504] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.533414 17505 ts_tablet_manager.cc:1428] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.032s user 0.010s sys 0.002s
I20260810 23:33:50.608906 17088 tablet_service.cc:1467] Processing CreateTablet for tablet c69104e44b294270b299ca6de1cdfdc9 (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:33:50.610775 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:33:50.612355 17524 leader_election.cc:288] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 election: Requested vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:50.614195 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72d0f3db05b4aabb69a46a8bc88f61e" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2466b33eeb034688bcf670ff612d452a"
I20260810 23:33:50.615463 17088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c69104e44b294270b299ca6de1cdfdc9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:50.615702 17030 leader_election.cc:341] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:50.620208 17303 tablet_service.cc:1467] Processing CreateTablet for tablet c69104e44b294270b299ca6de1cdfdc9 (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:50.621488 17303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c69104e44b294270b299ca6de1cdfdc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:50.631322 16327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Sending DeleteTablet for 3 replicas of tablet d31ab1862b084678b90352f9c8e242e3
I20260810 23:33:50.633072 16327 catalog_manager.cc:6565] ReplaceTablet: tablet d31ab1862b084678b90352f9c8e242e3 deleted and replaced by tablet ff16e0dc5dd24364890f95e1786b8b4d
I20260810 23:33:50.635859 17508 tablet_bootstrap.cc:492] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:33:50.637034 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet d31ab1862b084678b90352f9c8e242e3 with delete_type TABLET_DATA_DELETED (replaced by tablet ff16e0dc5dd24364890f95e1786b8b4d) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:50.640344 17091 tablet_service.cc:1514] Processing DeleteTablet for tablet d31ab1862b084678b90352f9c8e242e3 with delete_type TABLET_DATA_DELETED (replaced by tablet ff16e0dc5dd24364890f95e1786b8b4d) from {username='slave'} at 127.0.0.1:50894
I20260810 23:33:50.641458 17522 tablet_replica.cc:330] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:33:50.642374 17522 raft_consensus.cc:2229] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:50.642949 17522 raft_consensus.cc:2258] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:50.643374 17520 log.cc:1196] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/9ddfe4ea25ca44db94cbee975cce1334
I20260810 23:33:50.645820 17522 ts_tablet_manager.cc:1905] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:50.606113 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72d0f3db05b4aabb69a46a8bc88f61e" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36"
I20260810 23:33:50.652429 16904 raft_consensus.cc:3037] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:50.652729 17520 ts_tablet_manager.cc:1939] T 9ddfe4ea25ca44db94cbee975cce1334 P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:33:50.656982 17515 tablet_bootstrap.cc:492] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:33:50.596515 16860 tablet_service.cc:1467] Processing CreateTablet for tablet c69104e44b294270b299ca6de1cdfdc9 (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:50.659821 17515 ts_tablet_manager.cc:1397] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.170s user 0.012s sys 0.003s
I20260810 23:33:50.660323 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c69104e44b294270b299ca6de1cdfdc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:50.658872 16855 tablet_service.cc:1514] Processing DeleteTablet for tablet d31ab1862b084678b90352f9c8e242e3 with delete_type TABLET_DATA_DELETED (replaced by tablet ff16e0dc5dd24364890f95e1786b8b4d) from {username='slave'} at 127.0.0.1:52952
I20260810 23:33:50.666596 16316 catalog_manager.cc:4841] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): tablet 9ddfe4ea25ca44db94cbee975cce1334 (table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]) successfully deleted
I20260810 23:33:50.671996 16904 raft_consensus.cc:2454] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a651570d6e459d98bcf97054e07921 in term 1.
W20260810 23:33:50.673126 16316 catalog_manager.cc:4821] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): delete failed for tablet d31ab1862b084678b90352f9c8e242e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d31ab1862b084678b90352f9c8e242e3 already in progress: creating tablet
I20260810 23:33:50.684274 17508 tablet_bootstrap.cc:654] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:50.686758 17031 leader_election.cc:302] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36; no voters: 2466b33eeb034688bcf670ff612d452a
I20260810 23:33:50.687604 17522 ts_tablet_manager.cc:1918] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
@ 0x7f03a7cc09a0 google::LogMessage::Flush() at ??:0
I20260810 23:33:50.688056 17522 log.cc:1196] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/d31ab1862b084678b90352f9c8e242e3
I20260810 23:33:50.688336 17524 raft_consensus.cc:2781] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:50.688961 17524 raft_consensus.cc:686] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 1 LEADER]: Becoming Leader. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Running, Role: LEADER
I20260810 23:33:50.689996 17524 consensus_queue.cc:237] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.691767 17522 ts_tablet_manager.cc:1939] T d31ab1862b084678b90352f9c8e242e3 P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:33:50.700461 17521 ts_tablet_manager.cc:1918] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:50.700984 17521 log.cc:1196] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/9ddfe4ea25ca44db94cbee975cce1334
I20260810 23:33:50.701756 17521 ts_tablet_manager.cc:1939] T 9ddfe4ea25ca44db94cbee975cce1334 P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:33:50.702745 16311 catalog_manager.cc:4841] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): tablet d31ab1862b084678b90352f9c8e242e3 (table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]) successfully deleted
I20260810 23:33:50.705037 17091 tablet_service.cc:1467] Processing CreateTablet for tablet ff16e0dc5dd24364890f95e1786b8b4d (DEFAULT_TABLE table=default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:50.706364 17091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff16e0dc5dd24364890f95e1786b8b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:50.717044 17303 tablet_service.cc:1467] Processing CreateTablet for tablet ff16e0dc5dd24364890f95e1786b8b4d (DEFAULT_TABLE table=default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:50.718369 17303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff16e0dc5dd24364890f95e1786b8b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:50.663884 17515 raft_consensus.cc:348] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.719530 17515 raft_consensus.cc:374] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:50.721784 16314 catalog_manager.cc:4841] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): tablet 9ddfe4ea25ca44db94cbee975cce1334 (table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]) successfully deleted
W20260810 23:33:50.722699 16314 catalog_manager.cc:4821] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): delete failed for tablet d31ab1862b084678b90352f9c8e242e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d31ab1862b084678b90352f9c8e242e3 already in progress: creating tablet
I20260810 23:33:50.719893 17515 raft_consensus.cc:729] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:33:50.727180 16855 tablet_service.cc:1467] Processing CreateTablet for tablet ff16e0dc5dd24364890f95e1786b8b4d (DEFAULT_TABLE table=default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:50.728585 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff16e0dc5dd24364890f95e1786b8b4d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 23:33:50.736366 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:33:50.736959 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:33:50.738179 17534 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:50.727759 17515 consensus_queue.cc:260] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.749339 17515 ts_tablet_manager.cc:1428] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.089s user 0.008s sys 0.000s
I20260810 23:33:50.759796 17505 tablet_bootstrap.cc:492] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:33:50.768465 17505 tablet_bootstrap.cc:654] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:50.772694 17515 tablet_bootstrap.cc:492] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:33:50.784348 17508 tablet_bootstrap.cc:492] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:33:50.784873 17508 ts_tablet_manager.cc:1397] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.149s user 0.011s sys 0.007s
I20260810 23:33:50.784214 16327 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33758:
name: "default.Table_6336ff4bf71e49799b4a84e71a2f9859"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:50.779898 16326 catalog_manager.cc:5495] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a651570d6e459d98bcf97054e07921 (127.11.226.130). New cstate: current_term: 1 leader_uuid: "80a651570d6e459d98bcf97054e07921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20260810 23:33:50.793537 16325 master_service.cc:946] ReplaceTablet: received request to replace tablet a72d0f3db05b4aabb69a46a8bc88f61e from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:50.795053 16860 tablet_service.cc:1514] Processing DeleteTablet for tablet d31ab1862b084678b90352f9c8e242e3 with delete_type TABLET_DATA_DELETED (replaced by tablet ff16e0dc5dd24364890f95e1786b8b4d) from {username='slave'} at 127.0.0.1:52952
I20260810 23:33:50.787900 17508 raft_consensus.cc:348] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.796476 17508 raft_consensus.cc:374] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:50.796792 17508 raft_consensus.cc:729] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:50.801311 17520 tablet_replica.cc:330] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:33:50.802246 17520 raft_consensus.cc:2229] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:50.802717 17520 raft_consensus.cc:2258] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:50.805552 17520 ts_tablet_manager.cc:1905] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:50.815847 17508 consensus_queue.cc:260] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.837638 17508 ts_tablet_manager.cc:1428] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.052s user 0.006s sys 0.002s
I20260810 23:33:50.848312 16326 master_service.cc:946] ReplaceTablet: received request to replace tablet a72d0f3db05b4aabb69a46a8bc88f61e from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:50.856256 17508 tablet_bootstrap.cc:492] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:33:50.878979 17508 tablet_bootstrap.cc:654] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:50.884155 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet a72d0f3db05b4aabb69a46a8bc88f61e with delete_type TABLET_DATA_DELETED (replaced by tablet 403e842c37fa438090748c7eb2fd96f6) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:50.885321 17521 tablet_replica.cc:330] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:33:50.885816 17505 tablet_bootstrap.cc:492] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:33:50.886361 17521 raft_consensus.cc:2229] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:50.886507 17505 ts_tablet_manager.cc:1397] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.127s user 0.012s sys 0.003s
I20260810 23:33:50.886967 17521 raft_consensus.cc:2258] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:50.888429 16325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Sending DeleteTablet for 3 replicas of tablet a72d0f3db05b4aabb69a46a8bc88f61e
I20260810 23:33:50.890270 16325 catalog_manager.cc:6565] ReplaceTablet: tablet a72d0f3db05b4aabb69a46a8bc88f61e deleted and replaced by tablet 403e842c37fa438090748c7eb2fd96f6
I20260810 23:33:50.892450 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet a72d0f3db05b4aabb69a46a8bc88f61e with delete_type TABLET_DATA_DELETED (replaced by tablet 403e842c37fa438090748c7eb2fd96f6) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:50.893388 16855 tablet_service.cc:1514] Processing DeleteTablet for tablet a72d0f3db05b4aabb69a46a8bc88f61e with delete_type TABLET_DATA_DELETED (replaced by tablet 403e842c37fa438090748c7eb2fd96f6) from {username='slave'} at 127.0.0.1:52952
I20260810 23:33:50.894553 17515 tablet_bootstrap.cc:654] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:50.894872 17091 tablet_service.cc:1514] Processing DeleteTablet for tablet a72d0f3db05b4aabb69a46a8bc88f61e with delete_type TABLET_DATA_DELETED (replaced by tablet 403e842c37fa438090748c7eb2fd96f6) from {username='slave'} at 127.0.0.1:50894
I20260810 23:33:50.895991 17522 tablet_replica.cc:330] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:33:50.896199 17521 ts_tablet_manager.cc:1905] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:50.896831 17522 raft_consensus.cc:2229] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:50.897847 17522 pending_rounds.cc:62] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921: Trying to abort 1 pending ops.
I20260810 23:33:50.898161 17522 pending_rounds.cc:68] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317114186559229952 op_type: NO_OP noop_request { }
I20260810 23:33:50.898765 17522 raft_consensus.cc:2866] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:33:50.899192 17522 raft_consensus.cc:2258] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:50.900141 17505 raft_consensus.cc:348] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.901333 17505 raft_consensus.cc:374] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:50.901713 17505 raft_consensus.cc:729] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:33:50.902549 17522 ts_tablet_manager.cc:1905] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:50.902688 17505 consensus_queue.cc:260] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
@ 0x7f03a7cc070f google::LogMessage::~LogMessage() at ??:0
I20260810 23:33:50.930737 17505 ts_tablet_manager.cc:1428] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.044s user 0.007s sys 0.003s
I20260810 23:33:50.932390 17505 tablet_bootstrap.cc:492] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:33:50.933945 17524 raft_consensus.cc:482] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:50.934466 17524 raft_consensus.cc:504] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.936551 17524 leader_election.cc:288] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:50.942766 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69104e44b294270b299ca6de1cdfdc9" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36" is_pre_election: true
W20260810 23:33:50.944847 17031 leader_election.cc:341] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:50.947332 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69104e44b294270b299ca6de1cdfdc9" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2466b33eeb034688bcf670ff612d452a" is_pre_election: true
I20260810 23:33:50.949141 17515 tablet_bootstrap.cc:492] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
W20260810 23:33:50.949375 17030 leader_election.cc:341] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:50.949687 17515 ts_tablet_manager.cc:1397] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.177s user 0.004s sys 0.010s
I20260810 23:33:50.949813 17030 leader_election.cc:302] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a651570d6e459d98bcf97054e07921; no voters: 2466b33eeb034688bcf670ff612d452a, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:33:50.951123 17524 raft_consensus.cc:2726] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:50.957089 17520 ts_tablet_manager.cc:1918] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:50.957557 17520 log.cc:1196] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/d31ab1862b084678b90352f9c8e242e3
I20260810 23:33:50.958284 17520 ts_tablet_manager.cc:1939] T d31ab1862b084678b90352f9c8e242e3 P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:33:50.970148 17520 tablet_replica.cc:330] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:33:50.970927 17520 raft_consensus.cc:2229] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:50.971089 16316 catalog_manager.cc:4841] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): tablet d31ab1862b084678b90352f9c8e242e3 (table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]) successfully deleted
I20260810 23:33:50.958587 17515 raft_consensus.cc:348] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.971892 17515 raft_consensus.cc:374] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:50.972268 17515 raft_consensus.cc:729] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:33:50.972630 17508 tablet_bootstrap.cc:492] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:33:50.973169 17508 ts_tablet_manager.cc:1397] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.117s user 0.007s sys 0.011s
I20260810 23:33:50.976599 17508 raft_consensus.cc:348] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.977471 17508 raft_consensus.cc:374] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:50.977742 17508 raft_consensus.cc:729] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:50.978688 17522 ts_tablet_manager.cc:1918] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:50.979156 17522 log.cc:1196] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/a72d0f3db05b4aabb69a46a8bc88f61e
I20260810 23:33:50.980006 17522 ts_tablet_manager.cc:1939] T a72d0f3db05b4aabb69a46a8bc88f61e P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:33:50.983520 16311 catalog_manager.cc:4841] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): tablet a72d0f3db05b4aabb69a46a8bc88f61e (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
I20260810 23:33:50.984637 17508 consensus_queue.cc:260] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:50.990185 17521 ts_tablet_manager.cc:1918] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:50.998921 16855 tablet_service.cc:1467] Processing CreateTablet for tablet 403e842c37fa438090748c7eb2fd96f6 (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:51.000418 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403e842c37fa438090748c7eb2fd96f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:50.973089 17515 consensus_queue.cc:260] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.012867 17091 tablet_service.cc:1467] Processing CreateTablet for tablet 403e842c37fa438090748c7eb2fd96f6 (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:51.014331 17091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403e842c37fa438090748c7eb2fd96f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:51.016966 17515 ts_tablet_manager.cc:1428] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.067s user 0.008s sys 0.000s
I20260810 23:33:51.023239 17515 tablet_bootstrap.cc:492] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:33:51.027053 17520 raft_consensus.cc:2258] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:51.014402 17505 tablet_bootstrap.cc:654] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:51.030968 17303 tablet_service.cc:1467] Processing CreateTablet for tablet 403e842c37fa438090748c7eb2fd96f6 (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:51.033610 17508 ts_tablet_manager.cc:1428] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.060s user 0.007s sys 0.001s
I20260810 23:33:51.034986 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet d31ab1862b084678b90352f9c8e242e3 with delete_type TABLET_DATA_DELETED (replaced by tablet ff16e0dc5dd24364890f95e1786b8b4d) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:51.059954 17303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403e842c37fa438090748c7eb2fd96f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:51.062412 17515 tablet_bootstrap.cc:654] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:51.059991 17508 tablet_bootstrap.cc:492] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:33:51.073004 17508 tablet_bootstrap.cc:654] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:51.082703 17521 log.cc:1196] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/a72d0f3db05b4aabb69a46a8bc88f61e
I20260810 23:33:51.082834 17505 tablet_bootstrap.cc:492] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:33:51.083611 17521 ts_tablet_manager.cc:1939] T a72d0f3db05b4aabb69a46a8bc88f61e P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:33:51.083398 17505 ts_tablet_manager.cc:1397] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.151s user 0.011s sys 0.003s
I20260810 23:33:51.086876 16314 catalog_manager.cc:4841] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): tablet a72d0f3db05b4aabb69a46a8bc88f61e (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
I20260810 23:33:51.087961 17505 raft_consensus.cc:348] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.096180 17520 ts_tablet_manager.cc:1905] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:51.101686 16325 master_service.cc:946] ReplaceTablet: received request to replace tablet a72d0f3db05b4aabb69a46a8bc88f61e from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:51.110682 16325 master_service.cc:946] ReplaceTablet: received request to replace tablet c69104e44b294270b299ca6de1cdfdc9 from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:51.125639 17505 raft_consensus.cc:374] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:51.126183 17505 raft_consensus.cc:729] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:33:51.145891 16325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Sending DeleteTablet for 3 replicas of tablet c69104e44b294270b299ca6de1cdfdc9
I20260810 23:33:51.148198 16860 tablet_service.cc:1514] Processing DeleteTablet for tablet c69104e44b294270b299ca6de1cdfdc9 with delete_type TABLET_DATA_DELETED (replaced by tablet 00591bb4bb904e28a4ddd774b27ce967) from {username='slave'} at 127.0.0.1:52952
I20260810 23:33:51.150018 17515 tablet_bootstrap.cc:492] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:33:51.150879 17088 tablet_service.cc:1514] Processing DeleteTablet for tablet c69104e44b294270b299ca6de1cdfdc9 with delete_type TABLET_DATA_DELETED (replaced by tablet 00591bb4bb904e28a4ddd774b27ce967) from {username='slave'} at 127.0.0.1:50894
I20260810 23:33:51.151758 17515 ts_tablet_manager.cc:1397] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.129s user 0.012s sys 0.003s
I20260810 23:33:51.151861 17522 tablet_replica.cc:330] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
W20260810 23:33:51.151660 16314 catalog_manager.cc:4808] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): delete failed for tablet a72d0f3db05b4aabb69a46a8bc88f61e because the tablet was not found. No further retry: Not found: Tablet not found: a72d0f3db05b4aabb69a46a8bc88f61e
I20260810 23:33:51.152462 17303 tablet_service.cc:1514] Processing DeleteTablet for tablet c69104e44b294270b299ca6de1cdfdc9 with delete_type TABLET_DATA_DELETED (replaced by tablet 00591bb4bb904e28a4ddd774b27ce967) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:51.152640 17522 raft_consensus.cc:2229] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:51.153138 17522 raft_consensus.cc:2258] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:51.153844 17521 tablet_replica.cc:330] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:33:51.155555 17505 consensus_queue.cc:260] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.161491 17522 ts_tablet_manager.cc:1905] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:51.162046 17505 ts_tablet_manager.cc:1428] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.077s user 0.006s sys 0.001s
I20260810 23:33:51.162953 16325 catalog_manager.cc:6565] ReplaceTablet: tablet c69104e44b294270b299ca6de1cdfdc9 deleted and replaced by tablet 00591bb4bb904e28a4ddd774b27ce967
I20260810 23:33:51.181243 17508 tablet_bootstrap.cc:492] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:33:51.181012 17515 raft_consensus.cc:348] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.181773 17508 ts_tablet_manager.cc:1397] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.122s user 0.011s sys 0.001s
I20260810 23:33:51.184492 17508 raft_consensus.cc:348] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.185204 17508 raft_consensus.cc:374] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:51.185444 17508 raft_consensus.cc:729] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:51.185534 17088 tablet_service.cc:1467] Processing CreateTablet for tablet 00591bb4bb904e28a4ddd774b27ce967 (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:51.188764 17515 raft_consensus.cc:374] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:51.189142 17515 raft_consensus.cc:729] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:33:51.189606 17088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00591bb4bb904e28a4ddd774b27ce967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:51.190869 17508 consensus_queue.cc:260] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.192363 17515 consensus_queue.cc:260] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.197772 16855 tablet_service.cc:1467] Processing CreateTablet for tablet 00591bb4bb904e28a4ddd774b27ce967 (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:51.199230 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00591bb4bb904e28a4ddd774b27ce967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:51.204813 17505 tablet_bootstrap.cc:492] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:33:51.209568 17534 raft_consensus.cc:482] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:51.210292 17534 raft_consensus.cc:504] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.217578 17534 leader_election.cc:288] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
I20260810 23:33:51.218369 17534 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:51.218909 17534 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:33:51.223081 17515 ts_tablet_manager.cc:1428] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.071s user 0.005s sys 0.001s
I20260810 23:33:51.223636 17521 raft_consensus.cc:2229] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:51.224218 17515 tablet_bootstrap.cc:492] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:33:51.224539 17126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69104e44b294270b299ca6de1cdfdc9" candidate_uuid: "2466b33eeb034688bcf670ff612d452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a651570d6e459d98bcf97054e07921" is_pre_election: true
I20260810 23:33:51.225852 17521 raft_consensus.cc:2258] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:51.228442 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69104e44b294270b299ca6de1cdfdc9" candidate_uuid: "2466b33eeb034688bcf670ff612d452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36" is_pre_election: true
I20260810 23:33:51.229212 16904 raft_consensus.cc:2454] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2466b33eeb034688bcf670ff612d452a in term 0.
W20260810 23:33:51.228149 17234 leader_election.cc:341] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Illegal state: Tablet not RUNNING: STOPPED
I20260810 23:33:51.210655 17303 tablet_service.cc:1467] Processing CreateTablet for tablet 00591bb4bb904e28a4ddd774b27ce967 (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
W20260810 23:33:51.233698 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:33:51.234122 17303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00591bb4bb904e28a4ddd774b27ce967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:51.237100 17237 leader_election.cc:302] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a, c3b587fa042b42e4b66bb113245c0e36; no voters: 80a651570d6e459d98bcf97054e07921
W20260810 23:33:51.239310 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:33:51.228617 17534 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
I20260810 23:33:51.240454 17534 raft_consensus.cc:2781] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:51.240818 17534 raft_consensus.cc:482] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:51.239936 17237 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:33:51.241138 17534 raft_consensus.cc:3037] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:51.231215 17508 ts_tablet_manager.cc:1428] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.049s user 0.008s sys 0.000s
I20260810 23:33:51.255892 17508 tablet_bootstrap.cc:492] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:33:51.268622 17505 tablet_bootstrap.cc:654] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:51.273588 17522 ts_tablet_manager.cc:1918] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:51.274106 17522 log.cc:1196] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/c69104e44b294270b299ca6de1cdfdc9
@ 0x7f03b2d5baea kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260810 23:33:51.277052 17522 ts_tablet_manager.cc:1939] T c69104e44b294270b299ca6de1cdfdc9 P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:33:51.279093 17515 tablet_bootstrap.cc:654] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:51.280694 16311 catalog_manager.cc:4841] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): tablet c69104e44b294270b299ca6de1cdfdc9 (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]) successfully deleted
I20260810 23:33:51.282227 17604 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:51.282950 17521 ts_tablet_manager.cc:1905] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:51.287544 17520 ts_tablet_manager.cc:1918] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:51.296347 17520 log.cc:1196] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/a72d0f3db05b4aabb69a46a8bc88f61e
I20260810 23:33:51.297230 17520 ts_tablet_manager.cc:1939] T a72d0f3db05b4aabb69a46a8bc88f61e P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:33:51.313234 16325 master_service.cc:946] ReplaceTablet: received request to replace tablet 403e842c37fa438090748c7eb2fd96f6 from {username='slave'} at 127.0.0.1:33758
@ 0x7f03b2d5b33c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260810 23:33:51.316988 16316 catalog_manager.cc:4841] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): tablet a72d0f3db05b4aabb69a46a8bc88f61e (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
I20260810 23:33:51.322947 17505 tablet_bootstrap.cc:492] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:33:51.323447 17505 ts_tablet_manager.cc:1397] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.119s user 0.004s sys 0.011s
I20260810 23:33:51.320416 17508 tablet_bootstrap.cc:654] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:51.326152 17505 raft_consensus.cc:348] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.326906 17505 raft_consensus.cc:374] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:51.327184 17505 raft_consensus.cc:729] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:33:51.328608 17534 raft_consensus.cc:504] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.336611 17520 tablet_replica.cc:330] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:33:51.337464 17520 raft_consensus.cc:2229] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:51.338006 17520 raft_consensus.cc:2258] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus is shut down!
@ 0x7f03b2d5efd2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260810 23:33:51.343909 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69104e44b294270b299ca6de1cdfdc9" candidate_uuid: "2466b33eeb034688bcf670ff612d452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36"
W20260810 23:33:51.345731 17237 leader_election.cc:341] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:33:51.347790 17515 tablet_bootstrap.cc:492] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:33:51.348263 17515 ts_tablet_manager.cc:1397] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.124s user 0.012s sys 0.000s
I20260810 23:33:51.351154 17515 raft_consensus.cc:348] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.353667 17520 ts_tablet_manager.cc:1905] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:33:51.353765 17234 leader_election.cc:341] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: c69104e44b294270b299ca6de1cdfdc9
I20260810 23:33:51.354198 17234 leader_election.cc:302] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:33:51.355079 17604 raft_consensus.cc:2726] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:33:51.355999 17534 leader_election.cc:288] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 election: Requested vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
I20260810 23:33:51.360102 16325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Sending DeleteTablet for 3 replicas of tablet 403e842c37fa438090748c7eb2fd96f6
I20260810 23:33:51.362133 16860 tablet_service.cc:1514] Processing DeleteTablet for tablet 403e842c37fa438090748c7eb2fd96f6 with delete_type TABLET_DATA_DELETED (replaced by tablet 781caf1dd25544b1a26093aafc740b8c) from {username='slave'} at 127.0.0.1:52952
I20260810 23:33:51.364961 17088 tablet_service.cc:1514] Processing DeleteTablet for tablet 403e842c37fa438090748c7eb2fd96f6 with delete_type TABLET_DATA_DELETED (replaced by tablet 781caf1dd25544b1a26093aafc740b8c) from {username='slave'} at 127.0.0.1:50894
I20260810 23:33:51.369189 17508 tablet_bootstrap.cc:492] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
@ 0x7f03b2d5ef8a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260810 23:33:51.369727 17508 ts_tablet_manager.cc:1397] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.114s user 0.009s sys 0.003s
W20260810 23:33:51.374370 16311 catalog_manager.cc:4821] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): delete failed for tablet 403e842c37fa438090748c7eb2fd96f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 403e842c37fa438090748c7eb2fd96f6 already in progress: creating tablet
I20260810 23:33:51.375072 16325 catalog_manager.cc:6565] ReplaceTablet: tablet 403e842c37fa438090748c7eb2fd96f6 deleted and replaced by tablet 781caf1dd25544b1a26093aafc740b8c
I20260810 23:33:51.375236 16326 master_service.cc:946] ReplaceTablet: received request to replace tablet ff16e0dc5dd24364890f95e1786b8b4d from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:51.360213 17505 consensus_queue.cc:260] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.385712 17505 ts_tablet_manager.cc:1428] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.062s user 0.004s sys 0.003s
I20260810 23:33:51.386788 17505 tablet_bootstrap.cc:492] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:33:51.379310 17508 raft_consensus.cc:348] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.351898 17515 raft_consensus.cc:374] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:51.388571 17515 raft_consensus.cc:729] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:33:51.388159 17508 raft_consensus.cc:374] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:51.390733 17508 raft_consensus.cc:729] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:51.391678 17508 consensus_queue.cc:260] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.394605 17521 ts_tablet_manager.cc:1918] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:51.394989 17521 log.cc:1196] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/d31ab1862b084678b90352f9c8e242e3
I20260810 23:33:51.395663 17521 ts_tablet_manager.cc:1939] T d31ab1862b084678b90352f9c8e242e3 P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:33:51.396461 17515 consensus_queue.cc:260] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.402999 17088 tablet_service.cc:1514] Processing DeleteTablet for tablet 403e842c37fa438090748c7eb2fd96f6 with delete_type TABLET_DATA_DELETED (replaced by tablet 781caf1dd25544b1a26093aafc740b8c) from {username='slave'} at 127.0.0.1:50894
I20260810 23:33:51.410320 17515 ts_tablet_manager.cc:1428] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.062s user 0.007s sys 0.000s
I20260810 23:33:51.410573 17522 tablet_replica.cc:330] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:33:51.411324 17515 tablet_bootstrap.cc:492] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:33:51.411310 17522 raft_consensus.cc:2229] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:51.411993 17522 raft_consensus.cc:2258] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:51.412873 17303 tablet_service.cc:1514] Processing DeleteTablet for tablet 403e842c37fa438090748c7eb2fd96f6 with delete_type TABLET_DATA_DELETED (replaced by tablet 781caf1dd25544b1a26093aafc740b8c) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:51.414696 17508 ts_tablet_manager.cc:1428] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20260810 23:33:51.421269 17522 ts_tablet_manager.cc:1905] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:51.431739 17508 tablet_bootstrap.cc:492] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:33:51.435081 16855 tablet_service.cc:1467] Processing CreateTablet for tablet 781caf1dd25544b1a26093aafc740b8c (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:51.436594 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 781caf1dd25544b1a26093aafc740b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:51.438763 17088 tablet_service.cc:1467] Processing CreateTablet for tablet 781caf1dd25544b1a26093aafc740b8c (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:51.440099 17088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 781caf1dd25544b1a26093aafc740b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:51.442204 17521 tablet_replica.cc:330] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:33:51.443043 17521 raft_consensus.cc:2229] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:51.443558 17521 raft_consensus.cc:2258] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:51.445292 17505 tablet_bootstrap.cc:654] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:51.451917 17521 ts_tablet_manager.cc:1905] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f03b2d5ef1a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260810 23:33:51.455794 17508 tablet_bootstrap.cc:654] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:51.458452 17515 tablet_bootstrap.cc:654] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:51.458858 16326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61534c7c271c4e3393ff5b46a3498f9b: Sending DeleteTablet for 3 replicas of tablet ff16e0dc5dd24364890f95e1786b8b4d
I20260810 23:33:51.461182 16860 tablet_service.cc:1514] Processing DeleteTablet for tablet ff16e0dc5dd24364890f95e1786b8b4d with delete_type TABLET_DATA_DELETED (replaced by tablet c4f50fab4316426d897292ff17acd27d) from {username='slave'} at 127.0.0.1:52952
I20260810 23:33:51.462709 16326 catalog_manager.cc:6565] ReplaceTablet: tablet ff16e0dc5dd24364890f95e1786b8b4d deleted and replaced by tablet c4f50fab4316426d897292ff17acd27d
I20260810 23:33:51.469830 17091 tablet_service.cc:1514] Processing DeleteTablet for tablet ff16e0dc5dd24364890f95e1786b8b4d with delete_type TABLET_DATA_DELETED (replaced by tablet c4f50fab4316426d897292ff17acd27d) from {username='slave'} at 127.0.0.1:50894
I20260810 23:33:51.472448 17531 raft_consensus.cc:482] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:51.473210 17531 raft_consensus.cc:504] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.476886 17508 tablet_bootstrap.cc:492] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:33:51.477337 17508 ts_tablet_manager.cc:1397] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.000s
I20260810 23:33:51.480691 17531 leader_election.cc:288] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517), 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
@ 0x7f03b2d5eee2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260810 23:33:51.497898 17508 raft_consensus.cc:348] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.498703 17508 raft_consensus.cc:374] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:51.500422 17508 raft_consensus.cc:729] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:51.508369 17508 consensus_queue.cc:260] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.513130 17520 ts_tablet_manager.cc:1918] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:51.513609 17520 log.cc:1196] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/c69104e44b294270b299ca6de1cdfdc9
@ 0x7f03b2d5e1de std::__1::__function::__func<>::operator()() at ??:0
@ 0x580785 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5805ba std::__1::function<>::operator()() at ??:?
I20260810 23:33:51.521402 17505 tablet_bootstrap.cc:492] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:33:51.522251 17505 ts_tablet_manager.cc:1397] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.136s user 0.010s sys 0.005s
I20260810 23:33:51.525024 17505 raft_consensus.cc:348] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.525776 17505 raft_consensus.cc:374] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:51.526053 17505 raft_consensus.cc:729] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:33:51.526818 17505 consensus_queue.cc:260] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.532846 17508 ts_tablet_manager.cc:1428] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.055s user 0.006s sys 0.001s
I20260810 23:33:51.534371 17515 tablet_bootstrap.cc:492] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:33:51.534876 17515 ts_tablet_manager.cc:1397] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.124s user 0.010s sys 0.004s
W20260810 23:33:51.536796 16314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.189:34939, user_credentials={real_user=slave}} blocked reactor thread for 132404us
I20260810 23:33:51.538712 17303 tablet_service.cc:1514] Processing DeleteTablet for tablet ff16e0dc5dd24364890f95e1786b8b4d with delete_type TABLET_DATA_DELETED (replaced by tablet c4f50fab4316426d897292ff17acd27d) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:51.540474 17303 tablet_service.cc:1514] Processing DeleteTablet for tablet ff16e0dc5dd24364890f95e1786b8b4d with delete_type TABLET_DATA_DELETED (replaced by tablet c4f50fab4316426d897292ff17acd27d) from {username='slave'} at 127.0.0.1:55552
I20260810 23:33:51.543010 17303 tablet_service.cc:1467] Processing CreateTablet for tablet 781caf1dd25544b1a26093aafc740b8c (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:51.544461 17303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 781caf1dd25544b1a26093aafc740b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:51.544543 17520 ts_tablet_manager.cc:1939] T c69104e44b294270b299ca6de1cdfdc9 P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
@ 0x7f03a9e71dc7 kudu::Thread::SuperviseThread() at ??:0
@ 0x4230f0 __tsan_thread_start_func at ??:?
I20260810 23:33:51.553486 17508 tablet_bootstrap.cc:492] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:33:51.554563 16314 catalog_manager.cc:4841] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): tablet d31ab1862b084678b90352f9c8e242e3 (table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]) successfully deleted
I20260810 23:33:51.555342 16316 catalog_manager.cc:4841] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): tablet c69104e44b294270b299ca6de1cdfdc9 (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]) successfully deleted
I20260810 23:33:51.556917 17520 tablet_replica.cc:330] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:33:51.557950 17520 raft_consensus.cc:2229] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:51.558472 17520 raft_consensus.cc:2258] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:51.563526 17522 ts_tablet_manager.cc:1918] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:51.565049 17515 raft_consensus.cc:348] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.566181 17515 raft_consensus.cc:374] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:51.566620 17515 raft_consensus.cc:729] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:33:51.567562 17521 ts_tablet_manager.cc:1918] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:51.568075 17521 log.cc:1196] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/c69104e44b294270b299ca6de1cdfdc9
I20260810 23:33:51.568871 17521 ts_tablet_manager.cc:1939] T c69104e44b294270b299ca6de1cdfdc9 P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:33:51.571144 17524 raft_consensus.cc:482] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:51.571660 17524 raft_consensus.cc:504] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.574584 17508 tablet_bootstrap.cc:654] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:51.574644 17524 leader_election.cc:288] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:51.576406 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff16e0dc5dd24364890f95e1786b8b4d" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2466b33eeb034688bcf670ff612d452a" is_pre_election: true
I20260810 23:33:51.576442 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff16e0dc5dd24364890f95e1786b8b4d" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36" is_pre_election: true
I20260810 23:33:51.577200 16904 raft_consensus.cc:2454] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a651570d6e459d98bcf97054e07921 in term 0.
I20260810 23:33:51.577193 17341 raft_consensus.cc:2454] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a651570d6e459d98bcf97054e07921 in term 0.
I20260810 23:33:51.577368 17505 ts_tablet_manager.cc:1428] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.055s user 0.005s sys 0.001s
I20260810 23:33:51.581347 17030 leader_election.cc:302] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a, 80a651570d6e459d98bcf97054e07921; no voters:
I20260810 23:33:51.582681 17524 raft_consensus.cc:2781] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:51.583230 17524 raft_consensus.cc:482] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:51.583616 17524 raft_consensus.cc:3037] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:51.585897 17520 ts_tablet_manager.cc:1905] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:51.587210 16326 master_service.cc:946] ReplaceTablet: received request to replace tablet 403e842c37fa438090748c7eb2fd96f6 from {username='slave'} at 127.0.0.1:33758
I20260810 23:33:51.567575 17515 consensus_queue.cc:260] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.593923 17524 raft_consensus.cc:504] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
@ 0x7f03ab4df6db start_thread at ??:0
I20260810 23:33:51.600090 17522 log.cc:1196] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/403e842c37fa438090748c7eb2fd96f6
I20260810 23:33:51.601099 17522 ts_tablet_manager.cc:1939] T 403e842c37fa438090748c7eb2fd96f6 P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:33:51.603871 17521 tablet_replica.cc:330] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:33:51.604291 16311 catalog_manager.cc:4841] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): tablet 403e842c37fa438090748c7eb2fd96f6 (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
I20260810 23:33:51.604795 17521 raft_consensus.cc:2229] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:51.604784 16314 catalog_manager.cc:4841] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): tablet c69104e44b294270b299ca6de1cdfdc9 (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]) successfully deleted
I20260810 23:33:51.605345 17515 ts_tablet_manager.cc:1428] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.070s user 0.009s sys 0.000s
I20260810 23:33:51.605327 17521 raft_consensus.cc:2258] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:51.606467 17515 tablet_bootstrap.cc:492] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:33:51.608060 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff16e0dc5dd24364890f95e1786b8b4d" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2466b33eeb034688bcf670ff612d452a"
I20260810 23:33:51.608793 17341 raft_consensus.cc:3037] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:51.613368 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff16e0dc5dd24364890f95e1786b8b4d" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36"
I20260810 23:33:51.614193 16904 raft_consensus.cc:3037] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:51.616591 17522 tablet_replica.cc:330] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:33:51.617664 17341 raft_consensus.cc:2454] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a651570d6e459d98bcf97054e07921 in term 1.
I20260810 23:33:51.619735 17030 leader_election.cc:302] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a, 80a651570d6e459d98bcf97054e07921; no voters:
I20260810 23:33:51.625859 16904 raft_consensus.cc:2454] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a651570d6e459d98bcf97054e07921 in term 1.
I20260810 23:33:51.626093 17625 raft_consensus.cc:2781] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:51.627863 17524 leader_election.cc:288] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 election: Requested vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:51.630609 17625 raft_consensus.cc:686] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 1 LEADER]: Becoming Leader. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Running, Role: LEADER
I20260810 23:33:51.631572 17625 consensus_queue.cc:237] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.635739 17515 tablet_bootstrap.cc:654] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:51.638284 17521 ts_tablet_manager.cc:1905] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:51.646533 17508 tablet_bootstrap.cc:492] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:33:51.647043 17508 ts_tablet_manager.cc:1397] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.094s user 0.011s sys 0.001s
@ 0x7f03a696561f clone at ??:0
I20260810 23:33:51.654439 17505 tablet_bootstrap.cc:492] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:33:51.655783 17524 raft_consensus.cc:482] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:51.664510 17505 tablet_bootstrap.cc:654] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:51.668435 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff16e0dc5dd24364890f95e1786b8b4d" candidate_uuid: "c3b587fa042b42e4b66bb113245c0e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2466b33eeb034688bcf670ff612d452a" is_pre_election: true
I20260810 23:33:51.668870 17126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff16e0dc5dd24364890f95e1786b8b4d" candidate_uuid: "c3b587fa042b42e4b66bb113245c0e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a651570d6e459d98bcf97054e07921" is_pre_election: true
I20260810 23:33:51.669451 17341 raft_consensus.cc:2379] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3b587fa042b42e4b66bb113245c0e36 in current term 1: Already voted for candidate 80a651570d6e459d98bcf97054e07921 in this term.
I20260810 23:33:51.671476 16797 leader_election.cc:302] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b587fa042b42e4b66bb113245c0e36; no voters: 2466b33eeb034688bcf670ff612d452a, 80a651570d6e459d98bcf97054e07921
I20260810 23:33:51.674777 17531 raft_consensus.cc:2726] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:51.656764 17524 raft_consensus.cc:504] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.681840 17520 ts_tablet_manager.cc:1918] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:51.681823 17524 leader_election.cc:288] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:51.682307 17520 log.cc:1196] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/403e842c37fa438090748c7eb2fd96f6
I20260810 23:33:51.683179 17520 ts_tablet_manager.cc:1939] T 403e842c37fa438090748c7eb2fd96f6 P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:33:51.682897 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00591bb4bb904e28a4ddd774b27ce967" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36" is_pre_election: true
I20260810 23:33:51.683729 16904 raft_consensus.cc:2454] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a651570d6e459d98bcf97054e07921 in term 0.
I20260810 23:33:51.685148 17031 leader_election.cc:302] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36; no voters:
I20260810 23:33:51.685832 17515 tablet_bootstrap.cc:492] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:33:51.686990 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00591bb4bb904e28a4ddd774b27ce967" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2466b33eeb034688bcf670ff612d452a" is_pre_election: true
I20260810 23:33:51.686915 17508 raft_consensus.cc:348] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.686302 17515 ts_tablet_manager.cc:1397] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.003s
I20260810 23:33:51.687830 17508 raft_consensus.cc:374] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:51.688161 17508 raft_consensus.cc:729] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:51.688684 17524 raft_consensus.cc:2781] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:51.689107 17524 raft_consensus.cc:482] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:51.689009 17508 consensus_queue.cc:260] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.689602 17524 raft_consensus.cc:3037] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:51.689949 17341 raft_consensus.cc:2454] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a651570d6e459d98bcf97054e07921 in term 0.
I20260810 23:33:51.691187 17515 raft_consensus.cc:348] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.691900 17515 raft_consensus.cc:374] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:51.692176 17515 raft_consensus.cc:729] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:33:51.693387 17515 consensus_queue.cc:260] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.697330 17515 ts_tablet_manager.cc:1428] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260810 23:33:51.700673 17520 tablet_replica.cc:330] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:33:51.701391 17520 raft_consensus.cc:2229] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:51.709286 17508 ts_tablet_manager.cc:1428] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.062s user 0.006s sys 0.004s
I20260810 23:33:51.710299 17520 raft_consensus.cc:2258] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:51.712982 17520 ts_tablet_manager.cc:1905] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:51.713845 17508 tablet_bootstrap.cc:492] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:33:51.719982 17505 tablet_bootstrap.cc:492] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:33:51.721218 17505 ts_tablet_manager.cc:1397] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.067s user 0.013s sys 0.000s
I20260810 23:33:51.724357 17505 raft_consensus.cc:348] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.725311 17505 raft_consensus.cc:374] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:51.725605 17505 raft_consensus.cc:729] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:33:51.726447 17505 consensus_queue.cc:260] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.734668 17508 tablet_bootstrap.cc:654] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:51.740336 17522 raft_consensus.cc:2229] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:51.752497 17522 pending_rounds.cc:62] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921: Trying to abort 1 pending ops.
I20260810 23:33:51.752807 17522 pending_rounds.cc:68] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317114190453223424 op_type: NO_OP noop_request { }
I20260810 23:33:51.753216 17522 raft_consensus.cc:2866] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:33:51.753541 17522 raft_consensus.cc:2258] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:51.756531 17522 ts_tablet_manager.cc:1905] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:51.739961 17524 raft_consensus.cc:504] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.762882 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00591bb4bb904e28a4ddd774b27ce967" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36"
I20260810 23:33:51.763571 16904 raft_consensus.cc:3037] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:51.772785 16904 raft_consensus.cc:2454] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a651570d6e459d98bcf97054e07921 in term 1.
I20260810 23:33:51.775548 17508 tablet_bootstrap.cc:492] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:33:51.776124 17508 ts_tablet_manager.cc:1397] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.008s
I20260810 23:33:51.778862 17508 raft_consensus.cc:348] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.740581 17505 ts_tablet_manager.cc:1428] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.019s user 0.010s sys 0.001s
I20260810 23:33:51.782934 17031 leader_election.cc:302] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36; no voters:
I20260810 23:33:51.785331 17625 raft_consensus.cc:2781] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:51.785892 17625 raft_consensus.cc:686] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [term 1 LEADER]: Becoming Leader. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Running, Role: LEADER
I20260810 23:33:51.786829 17625 consensus_queue.cc:237] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.789705 17524 leader_election.cc:288] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 election: Requested vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:51.791313 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00591bb4bb904e28a4ddd774b27ce967" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2466b33eeb034688bcf670ff612d452a"
I20260810 23:33:51.792255 17341 raft_consensus.cc:3037] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:51.793653 17520 ts_tablet_manager.cc:1918] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:51.794056 17520 log.cc:1196] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/ff16e0dc5dd24364890f95e1786b8b4d
I20260810 23:33:51.794714 17520 ts_tablet_manager.cc:1939] T ff16e0dc5dd24364890f95e1786b8b4d P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:33:51.820658 17341 raft_consensus.cc:2454] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a651570d6e459d98bcf97054e07921 in term 1.
I20260810 23:33:51.830456 17521 ts_tablet_manager.cc:1918] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:51.830945 17521 log.cc:1196] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/403e842c37fa438090748c7eb2fd96f6
I20260810 23:33:51.831722 17521 ts_tablet_manager.cc:1939] T 403e842c37fa438090748c7eb2fd96f6 P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:33:51.779903 17508 raft_consensus.cc:374] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:51.832831 17508 raft_consensus.cc:729] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:51.833758 17522 ts_tablet_manager.cc:1918] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:51.834167 17522 log.cc:1196] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/ff16e0dc5dd24364890f95e1786b8b4d
I20260810 23:33:51.834885 17522 ts_tablet_manager.cc:1939] T ff16e0dc5dd24364890f95e1786b8b4d P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:33:51.834477 17508 consensus_queue.cc:260] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.843571 17531 raft_consensus.cc:482] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:51.844129 17531 raft_consensus.cc:504] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.846239 17531 leader_election.cc:288] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517), 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:51.848587 17508 ts_tablet_manager.cc:1428] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.072s user 0.007s sys 0.000s
I20260810 23:33:51.849216 17126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "781caf1dd25544b1a26093aafc740b8c" candidate_uuid: "c3b587fa042b42e4b66bb113245c0e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a651570d6e459d98bcf97054e07921" is_pre_election: true
I20260810 23:33:51.849985 17126 raft_consensus.cc:2454] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b587fa042b42e4b66bb113245c0e36 in term 0.
I20260810 23:33:51.851703 16794 leader_election.cc:302] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36; no voters:
I20260810 23:33:51.852718 17531 raft_consensus.cc:2781] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:51.853061 17531 raft_consensus.cc:482] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:51.853366 17531 raft_consensus.cc:3037] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:51.855075 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "781caf1dd25544b1a26093aafc740b8c" candidate_uuid: "c3b587fa042b42e4b66bb113245c0e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2466b33eeb034688bcf670ff612d452a" is_pre_election: true
I20260810 23:33:51.855815 17341 raft_consensus.cc:2454] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b587fa042b42e4b66bb113245c0e36 in term 0.
I20260810 23:33:51.863518 17531 raft_consensus.cc:504] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:51.865190 17521 tablet_replica.cc:330] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:33:51.866101 17521 raft_consensus.cc:2229] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:51.866645 17521 raft_consensus.cc:2258] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:51.874650 17521 ts_tablet_manager.cc:1905] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:51.882423 17126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "781caf1dd25544b1a26093aafc740b8c" candidate_uuid: "c3b587fa042b42e4b66bb113245c0e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a651570d6e459d98bcf97054e07921"
I20260810 23:33:51.883095 17126 raft_consensus.cc:3037] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:51.891103 17126 raft_consensus.cc:2454] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b587fa042b42e4b66bb113245c0e36 in term 1.
I20260810 23:33:51.892450 16794 leader_election.cc:302] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36; no voters:
I20260810 23:33:51.899677 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "781caf1dd25544b1a26093aafc740b8c" candidate_uuid: "c3b587fa042b42e4b66bb113245c0e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2466b33eeb034688bcf670ff612d452a"
I20260810 23:33:51.900383 17341 raft_consensus.cc:3037] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:51.901206 17521 ts_tablet_manager.cc:1918] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:33:51.901695 17521 log.cc:1196] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/ff16e0dc5dd24364890f95e1786b8b4d
I20260810 23:33:51.902582 17521 ts_tablet_manager.cc:1939] T ff16e0dc5dd24364890f95e1786b8b4d P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:33:51.907366 17341 raft_consensus.cc:2454] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b587fa042b42e4b66bb113245c0e36 in term 1.
I20260810 23:33:51.907634 17531 leader_election.cc:288] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [CANDIDATE]: Term 1 election: Requested vote from peers 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517), 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:51.918256 17531 raft_consensus.cc:2781] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:51.918854 17531 raft_consensus.cc:686] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [term 1 LEADER]: Becoming Leader. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Running, Role: LEADER
I20260810 23:33:51.929571 17531 consensus_queue.cc:237] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:52.177014 17661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61534c7c271c4e3393ff5b46a3498f9b)
I20260810 23:33:52.177722 17661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:52.179936 17661 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:33:52.188148 16559 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
W20260810 23:33:52.325054 17234 connection.cc:535] server connection from 127.0.0.1:55552 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:52.326316 17428 connection.cc:535] client connection to 127.11.226.190:42189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:52.328945 17428 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.226.190:42189)
W20260810 23:33:52.330154 17428 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.226.190:42189)
W20260810 23:33:52.331761 17428 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.226.190:42189)
W20260810 23:33:52.332777 17236 connection.cc:535] client connection to 127.11.226.190:42189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:52.333462 16558 connection.cc:535] server connection from 127.0.0.1:51312 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:52.336151 16559 connection.cc:535] client connection to 127.11.226.190:42189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:52.339077 17029 connection.cc:535] server connection from 127.0.0.1:50894 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:52.339025 16794 connection.cc:535] server connection from 127.0.0.1:52952 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:52.341958 17619 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:52.339826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.226.190:42189 (local address 127.11.226.129:48857)
0810 23:33:52.340899 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:52.341126 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":460}
W20260810 23:33:52.341941 17578 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:52.334810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.226.190:42189 (local address 127.11.226.131:51325)
0810 23:33:52.336954 (+ 2144us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:52.337027 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:52.337431 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:52.340301 (+ 2870us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1498}
W20260810 23:33:52.343262 17664 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:52.334565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.226.190:42189 (local address 127.11.226.130:37433)
0810 23:33:52.335385 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:52.335435 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:52.340591 (+ 5156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":358}
W20260810 23:33:52.346309 17428 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.226.190:42189)
W20260810 23:33:52.346364 16559 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
I20260810 23:33:52.346875 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:52.347986 17661 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 23:33:52.788954 17670 consensus_queue.cc:1035] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260810 23:33:52.824100 17673 consensus_queue.cc:1035] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:33:52.825388 17670 consensus_queue.cc:1035] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:33:52.879498 16794 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.226.130:42517, user_credentials={real_user=slave}} blocked reactor thread for 74881.8us
I20260810 23:33:53.012615 17687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61534c7c271c4e3393ff5b46a3498f9b)
I20260810 23:33:53.013413 17687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:53.035774 17687 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:33:53.035897 16559 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:33:53.040475 16559 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:33:53.041055 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:33:53.041854 17687 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 23:33:53.070638 17674 consensus_queue.cc:1035] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 23:33:50.448561 17572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:50.425971 17563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:53.415115 17568 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2987 milliseconds
I20260810 23:33:53.415349 17488 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 23:33:53.417375 17488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:53.421592 17488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:53.423172 17488 hybrid_clock.cc:648] HybridClock initialized: now 1786404833423097 us; error 102 us; skew 500 ppm
I20260810 23:33:53.424320 17488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:53.442193 17488 webserver.cc:460] Webserver started at http://127.11.226.189:39779/ using document root <none> and password file <none>
I20260810 23:33:53.443678 17488 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:53.444034 17488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:53.456691 17488 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.004s
I20260810 23:33:53.464005 17711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:53.465883 17488 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260810 23:33:53.466337 17488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/wal
uuid: "d0596cdb0a514cc08d7df0ceea528339"
format_stamp: "Formatted at 2026-08-10 23:33:35 on dist-test-slave-kvfs"
I20260810 23:33:53.470317 17488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:53.543828 17488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:53.546026 17488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:53.546600 17488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:53.671875 17488 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.189:34939
I20260810 23:33:53.672137 17800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.189:34939 every 8 connection(s)
I20260810 23:33:53.675531 17488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-1/data/info.pb
I20260810 23:33:53.683851 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 17488
I20260810 23:33:53.690408 17802 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:33:53.700933 17802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Bootstrap starting.
I20260810 23:33:53.730300 17802 log.cc:824] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:53.761176 17802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:33:53.762372 17802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Bootstrap complete.
I20260810 23:33:53.796577 17802 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:53.799625 17802 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0596cdb0a514cc08d7df0ceea528339, State: Initialized, Role: FOLLOWER
I20260810 23:33:53.800892 17802 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:53.806443 17811 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:53.807628 17811 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:53.808312 17802 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:33:53.863049 17829 catalog_manager.cc:1259] Loaded cluster ID: 82c629e15cdf4d9cb19224cd28fe2c13
I20260810 23:33:53.863907 17829 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: loading cluster ID for follower catalog manager: success
I20260810 23:33:53.883435 17829 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: acquiring CA information for follower catalog manager: success
I20260810 23:33:53.895576 17829 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:33:53.866197 17488 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:33:53.904765 17488 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:33:54.058322 16979 heartbeater.cc:347] Connected to a master server at 127.11.226.189:34939
I20260810 23:33:54.071121 17745 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" instance_seqno: 1786404823090495) as {username='slave'} at 127.11.226.129:39857; Asking this server to re-register.
I20260810 23:33:54.080147 16979 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:54.081633 17194 heartbeater.cc:347] Connected to a master server at 127.11.226.189:34939
I20260810 23:33:54.083586 17846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61534c7c271c4e3393ff5b46a3498f9b)
I20260810 23:33:54.084172 17846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:54.085777 17744 ts_manager.cc:194] Registered new tserver with Master: c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773)
I20260810 23:33:54.086534 17846 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
I20260810 23:33:54.090718 17745 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "80a651570d6e459d98bcf97054e07921" instance_seqno: 1786404825666854) as {username='slave'} at 127.11.226.130:50043; Asking this server to re-register.
I20260810 23:33:54.092934 17194 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:54.097921 17745 ts_manager.cc:194] Registered new tserver with Master: 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:33:54.104317 16559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20260810 23:33:54.111652 16559 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
I20260810 23:33:54.116997 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339" is_pre_election: true
I20260810 23:33:54.118278 17764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bfd786d28e24be390cdbbb4007290a7 in term 3.
I20260810 23:33:54.119746 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7, d0596cdb0a514cc08d7df0ceea528339; no voters: 61534c7c271c4e3393ff5b46a3498f9b
I20260810 23:33:54.120785 17846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 23:33:54.121150 17846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 61534c7c271c4e3393ff5b46a3498f9b)
I20260810 23:33:54.121488 17846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 4 FOLLOWER]: Advancing to term 5
I20260810 23:33:54.129933 17846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:54.132121 17846 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 election: Requested vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
I20260810 23:33:54.137310 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bfd786d28e24be390cdbbb4007290a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: false dest_uuid: "d0596cdb0a514cc08d7df0ceea528339"
I20260810 23:33:54.138049 17764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 3 FOLLOWER]: Advancing to term 5
W20260810 23:33:54.148685 16559 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
I20260810 23:33:54.158092 17764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bfd786d28e24be390cdbbb4007290a7 in term 5.
I20260810 23:33:54.160840 16559 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7, d0596cdb0a514cc08d7df0ceea528339; no voters: 61534c7c271c4e3393ff5b46a3498f9b
I20260810 23:33:54.162007 17846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 5 FOLLOWER]: Leader election won for term 5
I20260810 23:33:54.178133 17846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 5 LEADER]: Becoming Leader. State: Replica: 0bfd786d28e24be390cdbbb4007290a7, State: Running, Role: LEADER
I20260810 23:33:54.182120 17409 heartbeater.cc:347] Connected to a master server at 127.11.226.189:34939
I20260810 23:33:54.182124 17846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:54.182791 17409 heartbeater.cc:503] Master 127.11.226.189:34939 was elected leader, sending a full tablet report...
I20260810 23:33:54.188236 17745 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2466b33eeb034688bcf670ff612d452a" instance_seqno: 1786404828007913) as {username='slave'} at 127.11.226.131:40463; Asking this server to re-register.
I20260810 23:33:54.189390 17853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bfd786d28e24be390cdbbb4007290a7. Latest consensus state: current_term: 5 leader_uuid: "0bfd786d28e24be390cdbbb4007290a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:54.190399 17409 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:54.190565 17853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:54.194988 17745 ts_manager.cc:194] Registered new tserver with Master: 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:54.198800 17856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:33:54.211400 17856 catalog_manager.cc:669] Loaded metadata for table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]
I20260810 23:33:54.213690 17856 catalog_manager.cc:669] Loaded metadata for table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]
I20260810 23:33:54.215607 17856 catalog_manager.cc:669] Loaded metadata for table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]
I20260810 23:33:54.224624 17856 tablet_loader.cc:96] loaded metadata for tablet 00591bb4bb904e28a4ddd774b27ce967 (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d])
I20260810 23:33:54.227355 17856 tablet_loader.cc:96] loaded metadata for tablet 781caf1dd25544b1a26093aafc740b8c (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97])
I20260810 23:33:54.228888 17856 tablet_loader.cc:96] loaded metadata for tablet c4f50fab4316426d897292ff17acd27d (table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb])
I20260810 23:33:54.232199 17856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:33:54.238366 17856 catalog_manager.cc:1259] Loaded cluster ID: 82c629e15cdf4d9cb19224cd28fe2c13
I20260810 23:33:54.238741 17856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:33:54.246497 17856 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:33:54.252527 17856 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: Loaded TSK: 0
I20260810 23:33:54.254439 17856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:33:54.259982 17856 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260810 23:33:54.265633 17856 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260810 23:33:54.292351 17764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Refusing update from remote peer 0bfd786d28e24be390cdbbb4007290a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20260810 23:33:54.297185 17846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 6, Time since last communication: 0.001s
F20260810 23:33:54.298986 16655 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_bca6d3ee229b492798aa0ce50dd4c9f8
*** Check failure stack trace: ***
I20260810 23:33:54.311059 16578 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56184:
name: "default.Table_6336ff4bf71e49799b4a84e71a2f9859"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:54.315346 16572 master_service.cc:946] ReplaceTablet: received request to replace tablet 00591bb4bb904e28a4ddd774b27ce967 from {username='slave'} at 127.0.0.1:56184
I20260810 23:33:54.319097 16574 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56184:
table { table_name: "default.Table_bca6d3ee229b492798aa0ce50dd4c9f8" } new_table_name: "default.Table_f348d0e4ead44bf8a9ad5adee8b016f3" modify_external_catalogs: true
I20260810 23:33:54.322973 16573 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56184:
name: "default.Table_0124b56cffb54cc0b4c79411d0526c06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:54.324887 16559 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 -> Peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Couldn't send request to peer 61534c7c271c4e3393ff5b46a3498f9b. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:54.330286 16570 master_service.cc:946] ReplaceTablet: received request to replace tablet ff16e0dc5dd24364890f95e1786b8b4d from {username='slave'} at 127.0.0.1:56184
I20260810 23:33:54.333230 16567 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56184:
name: "default.Table_585e701700f14afbb2df4af6ca5e03f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:54.333636 16566 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56184:
name: "default.Table_bf8e0e4f072044999c5d464208e52921"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:54.334424 16565 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56184:
table { table_name: "default.Table_edb188d6a21846618b6fc3fe0927de7c" } new_table_name: "default.Table_90d3ba7d3f7f4808accd7856a23d4348" modify_external_catalogs: true
I20260810 23:33:54.335188 16569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56184:
table { table_name: "default.Table_edccecac3c7e4be4a5e3eab151d78907" } modify_external_catalogs: true
I20260810 23:33:54.336311 16569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56184:
table { table_name: "default.Table_edccecac3c7e4be4a5e3eab151d78907" } modify_external_catalogs: true
@ 0x7f10fc0619a0 google::LogMessage::Flush() at ??:0
@ 0x7f10fc06170f google::LogMessage::~LogMessage() at ??:0
@ 0x7f11070fcaea kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260810 23:33:54.418515 17858 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bfd786d28e24be390cdbbb4007290a7. Latest consensus state: current_term: 5 leader_uuid: "0bfd786d28e24be390cdbbb4007290a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:54.420703 17858 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:54.423184 17864 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "0bfd786d28e24be390cdbbb4007290a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:54.424181 17864 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:54.432893 17853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "0bfd786d28e24be390cdbbb4007290a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:54.433848 17853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: LEADER
@ 0x7f11070fc33c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260810 23:33:54.442339 17858 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "0bfd786d28e24be390cdbbb4007290a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:54.443209 17858 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:54.460176 16572 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: Sending DeleteTablet for 3 replicas of tablet 00591bb4bb904e28a4ddd774b27ce967
@ 0x7f11070fffd2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260810 23:33:54.498756 16572 catalog_manager.cc:6565] ReplaceTablet: tablet 00591bb4bb904e28a4ddd774b27ce967 deleted and replaced by tablet e7caeb9b9a8b4a5693c891a27837326a
I20260810 23:33:54.509104 16855 tablet_service.cc:1514] Processing DeleteTablet for tablet 00591bb4bb904e28a4ddd774b27ce967 with delete_type TABLET_DATA_DELETED (replaced by tablet e7caeb9b9a8b4a5693c891a27837326a) from {username='slave'} at 127.0.0.1:36454
I20260810 23:33:54.530686 17303 tablet_service.cc:1514] Processing DeleteTablet for tablet 00591bb4bb904e28a4ddd774b27ce967 with delete_type TABLET_DATA_DELETED (replaced by tablet e7caeb9b9a8b4a5693c891a27837326a) from {username='slave'} at 127.0.0.1:57846
I20260810 23:33:54.531317 17882 tablet_replica.cc:330] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:33:54.532398 17882 raft_consensus.cc:2229] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:54.533056 17882 raft_consensus.cc:2258] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:54.535476 17882 ts_tablet_manager.cc:1905] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f11070fff8a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260810 23:33:54.555639 16855 tablet_service.cc:1467] Processing CreateTablet for tablet e7caeb9b9a8b4a5693c891a27837326a (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:54.557173 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7caeb9b9a8b4a5693c891a27837326a. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f11070fff1a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260810 23:33:54.545189 17884 tablet_replica.cc:330] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:33:54.580408 17882 ts_tablet_manager.cc:1918] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:54.580917 17882 log.cc:1196] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/00591bb4bb904e28a4ddd774b27ce967
I20260810 23:33:54.581350 17884 raft_consensus.cc:2229] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:54.581842 17882 ts_tablet_manager.cc:1939] T 00591bb4bb904e28a4ddd774b27ce967 P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:33:54.582060 17884 raft_consensus.cc:2258] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:54.587085 16560 catalog_manager.cc:4841] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): tablet 00591bb4bb904e28a4ddd774b27ce967 (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]) successfully deleted
@ 0x7f11070ffee2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260810 23:33:54.601732 17303 tablet_service.cc:1467] Processing CreateTablet for tablet e7caeb9b9a8b4a5693c891a27837326a (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:54.607409 17303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7caeb9b9a8b4a5693c891a27837326a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:54.626040 17884 ts_tablet_manager.cc:1905] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f11070ff1de std::__1::__function::__func<>::operator()() at ??:0
@ 0x580785 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5805ba std::__1::function<>::operator()() at ??:?
I20260810 23:33:54.643942 17088 tablet_service.cc:1514] Processing DeleteTablet for tablet 00591bb4bb904e28a4ddd774b27ce967 with delete_type TABLET_DATA_DELETED (replaced by tablet e7caeb9b9a8b4a5693c891a27837326a) from {username='slave'} at 127.0.0.1:53968
I20260810 23:33:54.646363 17091 tablet_service.cc:1467] Processing CreateTablet for tablet e7caeb9b9a8b4a5693c891a27837326a (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:54.649221 16572 master_service.cc:946] ReplaceTablet: received request to replace tablet 781caf1dd25544b1a26093aafc740b8c from {username='slave'} at 127.0.0.1:56184
I20260810 23:33:54.652228 17091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7caeb9b9a8b4a5693c891a27837326a. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f10fe212dc7 kudu::Thread::SuperviseThread() at ??:0
@ 0x4230f0 __tsan_thread_start_func at ??:?
I20260810 23:33:54.679627 16572 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: Sending DeleteTablet for 3 replicas of tablet 781caf1dd25544b1a26093aafc740b8c
I20260810 23:33:54.685245 16860 tablet_service.cc:1514] Processing DeleteTablet for tablet 781caf1dd25544b1a26093aafc740b8c with delete_type TABLET_DATA_DELETED (replaced by tablet f35cb671e0ea436b9f155d4cc7f31bd8) from {username='slave'} at 127.0.0.1:36454
I20260810 23:33:54.687460 17882 tablet_replica.cc:330] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:33:54.688585 17882 raft_consensus.cc:2229] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:54.694731 17882 raft_consensus.cc:2258] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f10ff8806db start_thread at ??:0
I20260810 23:33:54.702147 17882 ts_tablet_manager.cc:1905] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:54.706718 17087 tablet_service.cc:1514] Processing DeleteTablet for tablet 781caf1dd25544b1a26093aafc740b8c with delete_type TABLET_DATA_DELETED (replaced by tablet f35cb671e0ea436b9f155d4cc7f31bd8) from {username='slave'} at 127.0.0.1:53968
I20260810 23:33:54.712309 17860 mvcc.cc:205] Tried to move back new op lower bound from 7317114201234472960 to 7317114178386219008. Current Snapshot: MvccSnapshot[applied={T|T < 7317114201234472960 or (T in {7317114201234472960})}]
I20260810 23:33:54.713165 16570 master_service.cc:946] ReplaceTablet: received request to replace tablet 781caf1dd25544b1a26093aafc740b8c from {username='slave'} at 127.0.0.1:56184
I20260810 23:33:54.717581 16572 catalog_manager.cc:6565] ReplaceTablet: tablet 781caf1dd25544b1a26093aafc740b8c deleted and replaced by tablet f35cb671e0ea436b9f155d4cc7f31bd8
I20260810 23:33:54.719003 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet 781caf1dd25544b1a26093aafc740b8c with delete_type TABLET_DATA_DELETED (replaced by tablet f35cb671e0ea436b9f155d4cc7f31bd8) from {username='slave'} at 127.0.0.1:57846
@ 0x7f10fad0661f clone at ??:0
I20260810 23:33:54.741335 17882 ts_tablet_manager.cc:1918] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:54.741806 17882 log.cc:1196] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/781caf1dd25544b1a26093aafc740b8c
I20260810 23:33:54.742673 17882 ts_tablet_manager.cc:1939] T 781caf1dd25544b1a26093aafc740b8c P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:33:54.765093 17902 tablet_replica.cc:330] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:33:54.767180 17902 raft_consensus.cc:2229] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:54.774053 17902 raft_consensus.cc:2258] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:54.779554 17884 ts_tablet_manager.cc:1918] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:54.785770 17902 ts_tablet_manager.cc:1905] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:54.794694 17884 log.cc:1196] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/00591bb4bb904e28a4ddd774b27ce967
I20260810 23:33:54.803480 17921 tablet_bootstrap.cc:492] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:33:54.805675 17884 ts_tablet_manager.cc:1939] T 00591bb4bb904e28a4ddd774b27ce967 P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:33:54.810231 17884 tablet_replica.cc:330] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:33:54.811133 17884 raft_consensus.cc:2229] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:54.811801 17884 raft_consensus.cc:2258] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:54.814173 17884 ts_tablet_manager.cc:1905] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:54.813585 17923 tablet_bootstrap.cc:492] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:33:54.833287 17917 tablet_bootstrap.cc:492] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:33:54.846822 17884 ts_tablet_manager.cc:1918] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:54.847345 17884 log.cc:1196] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/781caf1dd25544b1a26093aafc740b8c
I20260810 23:33:54.847704 17921 tablet_bootstrap.cc:654] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:54.853662 17917 tablet_bootstrap.cc:654] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:54.854135 17923 tablet_bootstrap.cc:654] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:54.859042 17884 ts_tablet_manager.cc:1939] T 781caf1dd25544b1a26093aafc740b8c P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:33:54.865782 17917 tablet_bootstrap.cc:492] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:33:54.866428 17917 ts_tablet_manager.cc:1397] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.000s
I20260810 23:33:54.874550 17921 tablet_bootstrap.cc:492] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:33:54.874570 17923 tablet_bootstrap.cc:492] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:33:54.875149 17923 ts_tablet_manager.cc:1397] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.005s
I20260810 23:33:54.869694 17917 raft_consensus.cc:348] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:54.878901 17923 raft_consensus.cc:348] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:54.880123 17923 raft_consensus.cc:374] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:54.880539 17921 ts_tablet_manager.cc:1397] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.010s
I20260810 23:33:54.880540 17923 raft_consensus.cc:729] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:33:54.881048 17917 raft_consensus.cc:374] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:54.881513 17923 consensus_queue.cc:260] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:54.882586 17917 raft_consensus.cc:729] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:33:54.883481 17193 heartbeater.cc:503] Master 127.11.226.188:46109 was elected leader, sending a full tablet report...
I20260810 23:33:54.891774 17917 consensus_queue.cc:260] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:54.896270 17923 ts_tablet_manager.cc:1428] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.021s user 0.009s sys 0.002s
I20260810 23:33:54.897080 16977 heartbeater.cc:503] Master 127.11.226.188:46109 was elected leader, sending a full tablet report...
I20260810 23:33:54.917526 17917 ts_tablet_manager.cc:1428] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.051s user 0.002s sys 0.006s
I20260810 23:33:54.911556 17921 raft_consensus.cc:348] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:54.915704 17902 ts_tablet_manager.cc:1918] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:54.919278 17902 log.cc:1196] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/00591bb4bb904e28a4ddd774b27ce967
I20260810 23:33:54.920008 17921 raft_consensus.cc:374] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:54.928179 17921 raft_consensus.cc:729] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:54.929183 17902 ts_tablet_manager.cc:1939] T 00591bb4bb904e28a4ddd774b27ce967 P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:33:54.930248 17921 consensus_queue.cc:260] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:54.938331 17902 tablet_replica.cc:330] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:33:54.939334 17902 raft_consensus.cc:2229] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:54.939669 17406 heartbeater.cc:503] Master 127.11.226.188:46109 was elected leader, sending a full tablet report...
I20260810 23:33:54.941691 17921 ts_tablet_manager.cc:1428] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.060s user 0.008s sys 0.000s
I20260810 23:33:54.942687 17902 raft_consensus.cc:2258] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:54.945565 17902 ts_tablet_manager.cc:1905] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:54.972815 17902 ts_tablet_manager.cc:1918] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:54.973495 17902 log.cc:1196] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/781caf1dd25544b1a26093aafc740b8c
I20260810 23:33:54.974618 17902 ts_tablet_manager.cc:1939] T 781caf1dd25544b1a26093aafc740b8c P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:33:55.067874 17928 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:55.068540 17928 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:33:55.071398 17928 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:33:55.075789 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:33:55.077139 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:33:55.077636 17237 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:33:55.080197 17928 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:55.119668 17928 raft_consensus.cc:482] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:55.121470 17928 raft_consensus.cc:504] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:55.125584 17126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7caeb9b9a8b4a5693c891a27837326a" candidate_uuid: "2466b33eeb034688bcf670ff612d452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a651570d6e459d98bcf97054e07921" is_pre_election: true
I20260810 23:33:55.126617 17126 raft_consensus.cc:2454] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2466b33eeb034688bcf670ff612d452a in term 0.
I20260810 23:33:55.130537 17234 leader_election.cc:302] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a, 80a651570d6e459d98bcf97054e07921; no voters:
I20260810 23:33:55.137892 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7caeb9b9a8b4a5693c891a27837326a" candidate_uuid: "2466b33eeb034688bcf670ff612d452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36" is_pre_election: true
I20260810 23:33:55.138756 16904 raft_consensus.cc:2454] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2466b33eeb034688bcf670ff612d452a in term 0.
I20260810 23:33:55.141002 17930 raft_consensus.cc:2781] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:55.141443 17930 raft_consensus.cc:482] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:55.141558 17928 leader_election.cc:288] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517), c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773)
I20260810 23:33:55.141826 17930 raft_consensus.cc:3037] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:55.152709 17930 raft_consensus.cc:504] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:55.156942 17126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7caeb9b9a8b4a5693c891a27837326a" candidate_uuid: "2466b33eeb034688bcf670ff612d452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a651570d6e459d98bcf97054e07921"
I20260810 23:33:55.157708 17126 raft_consensus.cc:3037] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:55.158757 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7caeb9b9a8b4a5693c891a27837326a" candidate_uuid: "2466b33eeb034688bcf670ff612d452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36"
I20260810 23:33:55.159879 16904 raft_consensus.cc:3037] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:55.168248 17126 raft_consensus.cc:2454] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2466b33eeb034688bcf670ff612d452a in term 1.
I20260810 23:33:55.169235 16904 raft_consensus.cc:2454] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2466b33eeb034688bcf670ff612d452a in term 1.
I20260810 23:33:55.170158 17234 leader_election.cc:302] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a, 80a651570d6e459d98bcf97054e07921; no voters:
I20260810 23:33:55.171345 17928 raft_consensus.cc:2781] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:55.173061 17930 leader_election.cc:288] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 election: Requested vote from peers 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517), c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773)
I20260810 23:33:55.183151 17928 raft_consensus.cc:686] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [term 1 LEADER]: Becoming Leader. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Running, Role: LEADER
I20260810 23:33:55.184477 17928 consensus_queue.cc:237] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:55.193646 17932 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0bfd786d28e24be390cdbbb4007290a7)
I20260810 23:33:55.194523 17932 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
W20260810 23:33:55.196086 17237 connection.cc:535] server connection from 127.0.0.1:57846 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:55.196461 17028 connection.cc:535] server connection from 127.0.0.1:53968 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:55.197666 17429 connection.cc:535] client connection to 127.11.226.188:46109 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:55.200536 17031 connection.cc:535] client connection to 127.11.226.188:46109 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:55.201942 16798 connection.cc:535] server connection from 127.0.0.1:36454 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:55.202606 16798 connection.cc:535] client connection to 127.11.226.188:46109 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:33:55.202566 17932 leader_election.cc:288] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:55.210048 17931 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:55.210825 17931 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
W20260810 23:33:55.212551 17237 connection.cc:535] client connection to 127.11.226.188:46109 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:33:55.212915 17938 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:55.199101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.226.188:46109 (local address 127.0.0.1:59702)
0810 23:33:55.206880 (+ 7779us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:55.207192 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":7201,"threads_started":1}
W20260810 23:33:55.218436 17720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection reset by peer (error 104)
I20260810 23:33:55.219295 17931 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:33:55.219786 17429 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.226.188:46109)
W20260810 23:33:55.220860 17429 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.226.188:46109)
W20260810 23:33:55.221691 17429 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.226.188:46109)
W20260810 23:33:55.221750 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:33:55.223042 17429 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.226.188:46109)
W20260810 23:33:55.223445 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:33:55.223953 17237 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:33:55.226305 17931 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:33:55.233235 17718 leader_election.cc:334] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:33:55.234058 17720 leader_election.cc:334] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109): Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
I20260810 23:33:55.236362 17720 leader_election.cc:302] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0596cdb0a514cc08d7df0ceea528339; no voters: 0bfd786d28e24be390cdbbb4007290a7, 61534c7c271c4e3393ff5b46a3498f9b
I20260810 23:33:55.237834 17932 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 23:33:55.930809 17950 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0bfd786d28e24be390cdbbb4007290a7)
I20260810 23:33:55.931450 17950 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:55.933961 17950 leader_election.cc:288] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
W20260810 23:33:55.948769 17720 leader_election.cc:334] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109): Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
W20260810 23:33:55.949656 17718 leader_election.cc:334] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
I20260810 23:33:55.950073 17718 leader_election.cc:302] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0596cdb0a514cc08d7df0ceea528339; no voters: 0bfd786d28e24be390cdbbb4007290a7, 61534c7c271c4e3393ff5b46a3498f9b
I20260810 23:33:55.950771 17950 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 23:33:56.116353 17961 consensus_queue.cc:1035] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:33:56.148897 17970 consensus_queue.cc:1035] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:33:56.760465 12170 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
/tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.188:46109
--webserver_interface=127.11.226.188
--webserver_port=46275
--builtin_ntp_servers=127.11.226.148:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--hive_metastore_uris=thrift://127.0.0.1:41643
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260810 23:33:56.865399 17982 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0bfd786d28e24be390cdbbb4007290a7)
I20260810 23:33:56.865934 17982 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:56.867846 17982 leader_election.cc:288] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
W20260810 23:33:56.874238 17718 leader_election.cc:334] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:33:56.874840 17720 leader_election.cc:334] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109): Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
I20260810 23:33:56.875265 17720 leader_election.cc:302] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0596cdb0a514cc08d7df0ceea528339; no voters: 0bfd786d28e24be390cdbbb4007290a7, 61534c7c271c4e3393ff5b46a3498f9b
I20260810 23:33:56.875967 17982 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 23:33:57.213790 17981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:57.214635 17981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:57.215200 17981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:57.264504 17981 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260810 23:33:57.264982 17981 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260810 23:33:57.265311 17981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:33:57.265743 17981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:57.266125 17981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:33:57.266436 17981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:33:57.314796 17981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.226.148:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41643
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.226.188:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.226.188
--webserver_port=46275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
build id 4579239
TSAN enabled
I20260810 23:33:57.316931 17981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:57.320288 17981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:57.348489 18006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:57.410885 18021 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0bfd786d28e24be390cdbbb4007290a7)
I20260810 23:33:57.411459 18021 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:57.413748 18021 leader_election.cc:288] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
W20260810 23:33:57.443821 17720 leader_election.cc:334] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109): Network error: Client connection negotiation failed: client connection to 127.11.226.188:46109: connect: Connection refused (error 111)
W20260810 23:33:57.444684 17718 leader_election.cc:334] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
I20260810 23:33:57.446549 17718 leader_election.cc:302] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0596cdb0a514cc08d7df0ceea528339; no voters: 0bfd786d28e24be390cdbbb4007290a7, 61534c7c271c4e3393ff5b46a3498f9b
I20260810 23:33:57.448467 18021 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260810 23:33:57.348492 18009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:57.352607 17981 server_base.cc:1034] running on GCE node
W20260810 23:33:57.349050 18007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:58.729391 17981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:58.733119 17981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:58.734609 17981 hybrid_clock.cc:648] HybridClock initialized: now 1786404838734573 us; error 64 us; skew 500 ppm
I20260810 23:33:58.735605 17981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:58.749393 17981 webserver.cc:460] Webserver started at http://127.11.226.188:46275/ using document root <none> and password file <none>
I20260810 23:33:58.751200 17981 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:58.751605 17981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:58.765024 17981 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.006s sys 0.001s
I20260810 23:33:58.772176 18049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:58.773726 17981 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:33:58.774199 17981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data,/tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/wal
uuid: "0bfd786d28e24be390cdbbb4007290a7"
format_stamp: "Formatted at 2026-08-10 23:33:37 on dist-test-slave-kvfs"
I20260810 23:33:58.778141 17981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:58.826753 17981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:58.829237 17981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:58.829841 17981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:58.912374 17488 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:33:58.912832 17488 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:33:58.929536 17981 rpc_server.cc:307] RPC server started. Bound to: 127.11.226.188:46109
I20260810 23:33:58.929719 18143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.226.188:46109 every 8 connection(s)
I20260810 23:33:58.933723 17981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/master-2/data/info.pb
I20260810 23:33:58.942715 12170 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskt0RZpf/build/tsan/bin/kudu as pid 17981
I20260810 23:33:58.949617 18145 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:33:58.959954 18145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: Bootstrap starting.
I20260810 23:33:58.995525 18145 log.cc:824] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:59.024425 17406 heartbeater.cc:347] Connected to a master server at 127.11.226.188:46109
I20260810 23:33:59.025077 17406 heartbeater.cc:503] Master 127.11.226.188:46109 was elected leader, sending a full tablet report...
I20260810 23:33:59.121716 18145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=20 ignored=0} mutations{seen=34 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:33:59.122763 18145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: Bootstrap complete.
I20260810 23:33:59.144553 18145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:59.147086 18145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bfd786d28e24be390cdbbb4007290a7, State: Initialized, Role: FOLLOWER
I20260810 23:33:59.148222 18145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 5.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:59.151135 18154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:59.151901 18154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:59.152585 18145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:33:59.227928 17193 heartbeater.cc:347] Connected to a master server at 127.11.226.188:46109
I20260810 23:33:59.228713 17193 heartbeater.cc:503] Master 127.11.226.188:46109 was elected leader, sending a full tablet report...
I20260810 23:33:59.185014 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:33:59.236088 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:33:59.239372 18077 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "80a651570d6e459d98bcf97054e07921" instance_seqno: 1786404825666854) as {username='slave'} at 127.11.226.130:56639; Asking this server to re-register.
I20260810 23:33:59.240679 18174 catalog_manager.cc:1259] Loaded cluster ID: 82c629e15cdf4d9cb19224cd28fe2c13
I20260810 23:33:59.240978 18174 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: loading cluster ID for follower catalog manager: success
I20260810 23:33:59.241660 17193 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:59.246330 18077 ts_manager.cc:194] Registered new tserver with Master: 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
I20260810 23:33:59.254585 18174 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: acquiring CA information for follower catalog manager: success
I20260810 23:33:59.262533 18174 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:33:59.266695 16977 heartbeater.cc:347] Connected to a master server at 127.11.226.188:46109
I20260810 23:33:59.267339 16977 heartbeater.cc:503] Master 127.11.226.188:46109 was elected leader, sending a full tablet report...
I20260810 23:33:59.271252 18077 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" instance_seqno: 1786404823090495) as {username='slave'} at 127.11.226.129:54131; Asking this server to re-register.
I20260810 23:33:59.272950 16977 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:59.276804 18077 ts_manager.cc:194] Registered new tserver with Master: c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773)
I20260810 23:33:59.450183 18183 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0bfd786d28e24be390cdbbb4007290a7)
I20260810 23:33:59.450711 18183 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:59.452668 18183 leader_election.cc:288] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
W20260810 23:33:59.461586 17718 leader_election.cc:334] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
I20260810 23:33:59.477193 18096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0596cdb0a514cc08d7df0ceea528339" candidate_term: 6 candidate_status { last_received { term: 5 index: 38 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7" is_pre_election: true
I20260810 23:33:59.478543 18096 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0596cdb0a514cc08d7df0ceea528339 in term 5.
I20260810 23:33:59.479956 17720 leader_election.cc:302] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7, d0596cdb0a514cc08d7df0ceea528339; no voters: 61534c7c271c4e3393ff5b46a3498f9b
I20260810 23:33:59.480873 18183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 23:33:59.481268 18183 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0bfd786d28e24be390cdbbb4007290a7)
I20260810 23:33:59.481653 18183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 5 FOLLOWER]: Advancing to term 6
I20260810 23:33:59.490029 18183 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:59.492347 18183 leader_election.cc:288] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 election: Requested vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), 0bfd786d28e24be390cdbbb4007290a7 (127.11.226.188:46109)
I20260810 23:33:59.493816 18096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0596cdb0a514cc08d7df0ceea528339" candidate_term: 6 candidate_status { last_received { term: 5 index: 38 } } ignore_live_leader: false dest_uuid: "0bfd786d28e24be390cdbbb4007290a7"
I20260810 23:33:59.494644 18096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 5 FOLLOWER]: Advancing to term 6
W20260810 23:33:59.499287 17718 leader_election.cc:334] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
I20260810 23:33:59.505394 18096 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0596cdb0a514cc08d7df0ceea528339 in term 6.
I20260810 23:33:59.509159 17720 leader_election.cc:302] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7, d0596cdb0a514cc08d7df0ceea528339; no voters: 61534c7c271c4e3393ff5b46a3498f9b
I20260810 23:33:59.510236 18183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 6 FOLLOWER]: Leader election won for term 6
I20260810 23:33:59.511844 18183 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [term 6 LEADER]: Becoming Leader. State: Replica: d0596cdb0a514cc08d7df0ceea528339, State: Running, Role: LEADER
I20260810 23:33:59.513049 18183 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 5.38, Last appended by leader: 38, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:33:59.518478 18191 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0596cdb0a514cc08d7df0ceea528339. Latest consensus state: current_term: 6 leader_uuid: "d0596cdb0a514cc08d7df0ceea528339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:59.519476 18191 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:59.521623 18194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:33:59.530781 18194 catalog_manager.cc:669] Loaded metadata for table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb]
I20260810 23:33:59.532444 18194 catalog_manager.cc:669] Loaded metadata for table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]
I20260810 23:33:59.534171 18194 catalog_manager.cc:669] Loaded metadata for table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]
I20260810 23:33:59.535975 17406 heartbeater.cc:347] Connected to a master server at 127.11.226.188:46109
I20260810 23:33:59.536659 17406 heartbeater.cc:503] Master 127.11.226.188:46109 was elected leader, sending a full tablet report...
I20260810 23:33:59.541872 18077 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2466b33eeb034688bcf670ff612d452a" instance_seqno: 1786404828007913) as {username='slave'} at 127.11.226.131:60797; Asking this server to re-register.
I20260810 23:33:59.543736 17406 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:59.544481 18194 tablet_loader.cc:96] loaded metadata for tablet c4f50fab4316426d897292ff17acd27d (table default.table_bca6d3ee229b492798aa0ce50dd4c9f8 [id=101fcdd599344d29bb0464af2be3c5cb])
I20260810 23:33:59.545516 18194 tablet_loader.cc:96] loaded metadata for tablet e7caeb9b9a8b4a5693c891a27837326a (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d])
I20260810 23:33:59.546712 18194 tablet_loader.cc:96] loaded metadata for tablet f35cb671e0ea436b9f155d4cc7f31bd8 (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97])
I20260810 23:33:59.548161 18194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:33:59.548724 18077 ts_manager.cc:194] Registered new tserver with Master: 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:33:59.552923 18194 catalog_manager.cc:1259] Loaded cluster ID: 82c629e15cdf4d9cb19224cd28fe2c13
I20260810 23:33:59.553330 18194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:33:59.558735 18194 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:33:59.563109 18194 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Loaded TSK: 0
I20260810 23:33:59.564656 18194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:33:59.568843 18194 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20260810 23:33:59.574671 18194 catalog_manager.cc:5717] Last processed Hive Metastore notification event ID: -1
I20260810 23:33:59.595144 18096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Refusing update from remote peer d0596cdb0a514cc08d7df0ceea528339: Log matching property violated. Preceding OpId in replica: term: 5 index: 38. Preceding OpId from leader: term: 6 index: 40. (index mismatch)
I20260810 23:33:59.599263 18191 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.001s
W20260810 23:33:59.600418 17718 consensus_peers.cc:487] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 -> Peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Couldn't send request to peer 61534c7c271c4e3393ff5b46a3498f9b. Status: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:33:59.615974 18154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0596cdb0a514cc08d7df0ceea528339. Latest consensus state: current_term: 6 leader_uuid: "d0596cdb0a514cc08d7df0ceea528339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:59.616856 18154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:59.621667 18183 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "d0596cdb0a514cc08d7df0ceea528339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:59.622763 18183 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:59.626185 18154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "d0596cdb0a514cc08d7df0ceea528339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } } }
I20260810 23:33:59.627108 18154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:33:59.667340 16855 tablet_service.cc:1467] Processing CreateTablet for tablet f35cb671e0ea436b9f155d4cc7f31bd8 (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:59.677838 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35cb671e0ea436b9f155d4cc7f31bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:59.686172 17303 tablet_service.cc:1467] Processing CreateTablet for tablet f35cb671e0ea436b9f155d4cc7f31bd8 (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:59.692301 17744 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48402:
table { table_name: "default.Table_edb188d6a21846618b6fc3fe0927de7c" } new_table_name: "default.Table_90d3ba7d3f7f4808accd7856a23d4348" modify_external_catalogs: true
I20260810 23:33:59.697058 17741 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48402:
name: "default.Table_bf8e0e4f072044999c5d464208e52921"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:59.687937 17303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35cb671e0ea436b9f155d4cc7f31bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:59.695916 17745 master_service.cc:946] ReplaceTablet: received request to replace tablet e7caeb9b9a8b4a5693c891a27837326a from {username='slave'} at 127.0.0.1:48402
I20260810 23:33:59.700336 17738 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48402:
name: "default.Table_585e701700f14afbb2df4af6ca5e03f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:59.701820 17737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48402:
table { table_name: "default.Table_edccecac3c7e4be4a5e3eab151d78907" } modify_external_catalogs: true
I20260810 23:33:59.709928 17737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48402:
table { table_name: "default.Table_edccecac3c7e4be4a5e3eab151d78907" } modify_external_catalogs: true
I20260810 23:33:59.710639 17734 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48402:
table { table_name: "default.Table_bca6d3ee229b492798aa0ce50dd4c9f8" } new_table_name: "default.Table_f348d0e4ead44bf8a9ad5adee8b016f3" modify_external_catalogs: true
I20260810 23:33:59.713315 17732 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48402:
name: "default.Table_0124b56cffb54cc0b4c79411d0526c06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:59.713860 17731 master_service.cc:946] ReplaceTablet: received request to replace tablet e7caeb9b9a8b4a5693c891a27837326a from {username='slave'} at 127.0.0.1:48402
F20260810 23:33:59.715808 17828 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_bca6d3ee229b492798aa0ce50dd4c9f8
*** Check failure stack trace: ***
I20260810 23:33:59.737037 18181 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:59.737759 18181 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:33:59.741117 18181 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:33:59.743537 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:33:59.745673 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:33:59.746177 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:33:59.740334 17730 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48402:
name: "default.Table_6336ff4bf71e49799b4a84e71a2f9859"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:33:59.760067 18181 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:33:59.764416 18218 tablet_bootstrap.cc:492] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:33:59.776065 18218 tablet_bootstrap.cc:654] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:59.784909 17745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Sending DeleteTablet for 3 replicas of tablet e7caeb9b9a8b4a5693c891a27837326a
I20260810 23:33:59.786341 18220 tablet_bootstrap.cc:492] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:33:59.787111 17745 catalog_manager.cc:6565] ReplaceTablet: tablet e7caeb9b9a8b4a5693c891a27837326a deleted and replaced by tablet 783e71cfbc904ffc9d19e5b9257709c5
I20260810 23:33:59.787998 16855 tablet_service.cc:1514] Processing DeleteTablet for tablet e7caeb9b9a8b4a5693c891a27837326a with delete_type TABLET_DATA_DELETED (replaced by tablet 783e71cfbc904ffc9d19e5b9257709c5) from {username='slave'} at 127.0.0.1:36456
I20260810 23:33:59.807825 17091 tablet_service.cc:1467] Processing CreateTablet for tablet f35cb671e0ea436b9f155d4cc7f31bd8 (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:59.809525 17091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f35cb671e0ea436b9f155d4cc7f31bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:59.813546 17088 tablet_service.cc:1514] Processing DeleteTablet for tablet e7caeb9b9a8b4a5693c891a27837326a with delete_type TABLET_DATA_DELETED (replaced by tablet 783e71cfbc904ffc9d19e5b9257709c5) from {username='slave'} at 127.0.0.1:53976
I20260810 23:33:59.788302 17303 tablet_service.cc:1514] Processing DeleteTablet for tablet e7caeb9b9a8b4a5693c891a27837326a with delete_type TABLET_DATA_DELETED (replaced by tablet 783e71cfbc904ffc9d19e5b9257709c5) from {username='slave'} at 127.0.0.1:57860
I20260810 23:33:59.828927 18221 tablet_replica.cc:330] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:33:59.830555 18221 raft_consensus.cc:2229] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:59.836493 18221 raft_consensus.cc:2258] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:59.842345 18220 tablet_bootstrap.cc:654] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:59.846312 16855 tablet_service.cc:1467] Processing CreateTablet for tablet 783e71cfbc904ffc9d19e5b9257709c5 (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:59.847812 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783e71cfbc904ffc9d19e5b9257709c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:59.848305 17304 tablet_service.cc:1467] Processing CreateTablet for tablet 783e71cfbc904ffc9d19e5b9257709c5 (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:59.849552 17304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783e71cfbc904ffc9d19e5b9257709c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:59.854199 17087 tablet_service.cc:1467] Processing CreateTablet for tablet 783e71cfbc904ffc9d19e5b9257709c5 (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:59.855571 17087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783e71cfbc904ffc9d19e5b9257709c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:59.856040 18218 tablet_bootstrap.cc:492] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:33:59.856546 18218 ts_tablet_manager.cc:1397] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.093s user 0.010s sys 0.017s
I20260810 23:33:59.839705 18221 ts_tablet_manager.cc:1905] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:59.870761 18227 tablet_replica.cc:330] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:33:59.871800 18227 raft_consensus.cc:2229] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:59.872581 18227 raft_consensus.cc:2258] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:59.874032 18228 tablet_replica.cc:330] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:33:59.874964 18228 raft_consensus.cc:2229] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:59.876273 18228 raft_consensus.cc:2258] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:59.882133 18218 raft_consensus.cc:348] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:59.885447 18220 tablet_bootstrap.cc:492] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:33:59.885968 18228 ts_tablet_manager.cc:1905] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:59.888705 18227 ts_tablet_manager.cc:1905] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f6e76dd89a0 google::LogMessage::Flush() at ??:0
I20260810 23:33:59.897861 18218 raft_consensus.cc:374] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:59.898284 18218 raft_consensus.cc:729] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:33:59.903996 18221 ts_tablet_manager.cc:1918] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:59.904562 18221 log.cc:1196] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/e7caeb9b9a8b4a5693c891a27837326a
I20260810 23:33:59.904460 18218 consensus_queue.cc:260] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:59.905788 18221 ts_tablet_manager.cc:1939] T e7caeb9b9a8b4a5693c891a27837326a P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:33:59.886056 18220 ts_tablet_manager.cc:1397] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.105s user 0.005s sys 0.011s
I20260810 23:33:59.910036 18220 raft_consensus.cc:348] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:59.910964 18220 raft_consensus.cc:374] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:59.911291 18220 raft_consensus.cc:729] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:59.913405 18220 consensus_queue.cc:260] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:59.924803 16979 heartbeater.cc:503] Master 127.11.226.189:34939 was elected leader, sending a full tablet report...
I20260810 23:33:59.926496 18220 ts_tablet_manager.cc:1428] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260810 23:33:59.929376 18218 ts_tablet_manager.cc:1428] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.072s user 0.003s sys 0.006s
I20260810 23:33:59.930440 18220 tablet_bootstrap.cc:492] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:33:59.943944 17720 catalog_manager.cc:4841] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): tablet e7caeb9b9a8b4a5693c891a27837326a (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]) successfully deleted
I20260810 23:33:59.944893 18218 tablet_bootstrap.cc:492] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:33:59.951844 18220 tablet_bootstrap.cc:654] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:59.953773 18218 tablet_bootstrap.cc:654] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:59.969235 17409 heartbeater.cc:503] Master 127.11.226.189:34939 was elected leader, sending a full tablet report...
I20260810 23:33:59.977089 18220 tablet_bootstrap.cc:492] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:33:59.977628 18220 ts_tablet_manager.cc:1397] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.047s user 0.014s sys 0.000s
W20260810 23:33:59.975948 17745 catalog_manager.cc:5328] Ignoring report from unknown tablet 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:33:59.978979 17745 catalog_manager.cc:5328] Ignoring report from unknown tablet 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:33:59.985361 18220 raft_consensus.cc:348] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:59.986488 18220 raft_consensus.cc:374] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:59.986840 18220 raft_consensus.cc:729] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:33:59.984005 18227 ts_tablet_manager.cc:1918] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:59.988314 18227 log.cc:1196] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/e7caeb9b9a8b4a5693c891a27837326a
I20260810 23:33:59.987793 18220 consensus_queue.cc:260] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:33:59.989393 18227 ts_tablet_manager.cc:1939] T e7caeb9b9a8b4a5693c891a27837326a P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:33:59.993942 18228 ts_tablet_manager.cc:1918] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:33:59.994547 18228 log.cc:1196] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/e7caeb9b9a8b4a5693c891a27837326a
I20260810 23:33:59.995985 18228 ts_tablet_manager.cc:1939] T e7caeb9b9a8b4a5693c891a27837326a P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:33:59.998358 18220 ts_tablet_manager.cc:1428] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260810 23:33:59.999577 18181 raft_consensus.cc:482] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:00.005556 18181 raft_consensus.cc:504] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.005939 17718 catalog_manager.cc:4841] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): tablet e7caeb9b9a8b4a5693c891a27837326a (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]) successfully deleted
I20260810 23:34:00.007686 18181 leader_election.cc:288] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
I20260810 23:34:00.008879 17716 catalog_manager.cc:4841] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): tablet e7caeb9b9a8b4a5693c891a27837326a (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]) successfully deleted
I20260810 23:34:00.009436 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783e71cfbc904ffc9d19e5b9257709c5" candidate_uuid: "2466b33eeb034688bcf670ff612d452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36" is_pre_election: true
I20260810 23:34:00.011171 17126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783e71cfbc904ffc9d19e5b9257709c5" candidate_uuid: "2466b33eeb034688bcf670ff612d452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a651570d6e459d98bcf97054e07921" is_pre_election: true
W20260810 23:34:00.011636 17237 leader_election.cc:341] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Illegal state: must be running to vote when last-logged opid is not known
W20260810 23:34:00.012797 17234 leader_election.cc:341] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Illegal state: must be running to vote when last-logged opid is not known
I20260810 23:34:00.013222 17234 leader_election.cc:302] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:34:00.023038 17729 master_service.cc:946] ReplaceTablet: received request to replace tablet 783e71cfbc904ffc9d19e5b9257709c5 from {username='slave'} at 127.0.0.1:48402
I20260810 23:34:00.024438 18181 raft_consensus.cc:2726] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:34:00.027940 17731 master_service.cc:946] ReplaceTablet: received request to replace tablet 783e71cfbc904ffc9d19e5b9257709c5 from {username='slave'} at 127.0.0.1:48402
I20260810 23:34:00.040184 18234 tablet_bootstrap.cc:492] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:34:00.049827 17729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Sending DeleteTablet for 3 replicas of tablet 783e71cfbc904ffc9d19e5b9257709c5
I20260810 23:34:00.052088 17729 catalog_manager.cc:6565] ReplaceTablet: tablet 783e71cfbc904ffc9d19e5b9257709c5 deleted and replaced by tablet d09b4ee48905498991a54e4c98e9381d
I20260810 23:34:00.054770 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet 783e71cfbc904ffc9d19e5b9257709c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d09b4ee48905498991a54e4c98e9381d) from {username='slave'} at 127.0.0.1:57860
I20260810 23:34:00.055899 18228 tablet_replica.cc:330] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:34:00.056806 18228 raft_consensus.cc:2229] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:34:00.057384 18228 raft_consensus.cc:2258] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:00.058028 17091 tablet_service.cc:1514] Processing DeleteTablet for tablet 783e71cfbc904ffc9d19e5b9257709c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d09b4ee48905498991a54e4c98e9381d) from {username='slave'} at 127.0.0.1:53976
I20260810 23:34:00.059942 16855 tablet_service.cc:1514] Processing DeleteTablet for tablet 783e71cfbc904ffc9d19e5b9257709c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d09b4ee48905498991a54e4c98e9381d) from {username='slave'} at 127.0.0.1:36456
W20260810 23:34:00.061656 17716 catalog_manager.cc:4821] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): delete failed for tablet 783e71cfbc904ffc9d19e5b9257709c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 783e71cfbc904ffc9d19e5b9257709c5 already in progress: creating tablet
W20260810 23:34:00.063153 17720 catalog_manager.cc:4821] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): delete failed for tablet 783e71cfbc904ffc9d19e5b9257709c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 783e71cfbc904ffc9d19e5b9257709c5 already in progress: creating tablet
I20260810 23:34:00.066241 18228 ts_tablet_manager.cc:1905] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:34:00.071274 17234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.130:42517, user_credentials={real_user=slave}} blocked reactor thread for 58501.3us
I20260810 23:34:00.106164 18218 tablet_bootstrap.cc:492] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:34:00.107402 18218 ts_tablet_manager.cc:1397] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.163s user 0.030s sys 0.060s
I20260810 23:34:00.113963 18228 ts_tablet_manager.cc:1918] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:34:00.114447 18228 log.cc:1196] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/783e71cfbc904ffc9d19e5b9257709c5
I20260810 23:34:00.115191 18228 ts_tablet_manager.cc:1939] T 783e71cfbc904ffc9d19e5b9257709c5 P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:34:00.118044 18218 raft_consensus.cc:348] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.121681 18218 raft_consensus.cc:374] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:00.122102 18218 raft_consensus.cc:729] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:34:00.123167 18218 consensus_queue.cc:260] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.129580 18234 tablet_bootstrap.cc:654] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:00.138264 18218 ts_tablet_manager.cc:1428] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20260810 23:34:00.164225 18234 tablet_bootstrap.cc:492] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:34:00.164763 18234 ts_tablet_manager.cc:1397] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.131s user 0.026s sys 0.016s
I20260810 23:34:00.167723 18234 raft_consensus.cc:348] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.168576 18234 raft_consensus.cc:374] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:00.168854 18234 raft_consensus.cc:729] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:34:00.169682 18234 consensus_queue.cc:260] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.195564 17194 heartbeater.cc:503] Master 127.11.226.189:34939 was elected leader, sending a full tablet report...
I20260810 23:34:00.195992 18234 ts_tablet_manager.cc:1428] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.031s user 0.006s sys 0.018s
I20260810 23:34:00.199461 18234 tablet_bootstrap.cc:492] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
@ 0x7f6e76dd870f google::LogMessage::~LogMessage() at ??:0
I20260810 23:34:00.209048 18234 tablet_bootstrap.cc:654] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:00.209404 17718 catalog_manager.cc:4841] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): tablet 783e71cfbc904ffc9d19e5b9257709c5 (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]) successfully deleted
W20260810 23:34:00.227257 17718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260810 23:34:00.233035 16855 tablet_service.cc:1514] Processing DeleteTablet for tablet 783e71cfbc904ffc9d19e5b9257709c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d09b4ee48905498991a54e4c98e9381d) from {username='slave'} at 127.0.0.1:36456
I20260810 23:34:00.233886 18221 tablet_replica.cc:330] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:34:00.233785 16860 tablet_service.cc:1467] Processing CreateTablet for tablet d09b4ee48905498991a54e4c98e9381d (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:34:00.234753 18221 raft_consensus.cc:2229] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:34:00.235277 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09b4ee48905498991a54e4c98e9381d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:34:00.235299 18221 raft_consensus.cc:2258] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:00.237322 17091 tablet_service.cc:1467] Processing CreateTablet for tablet d09b4ee48905498991a54e4c98e9381d (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:34:00.238686 17091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09b4ee48905498991a54e4c98e9381d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:34:00.238947 18221 ts_tablet_manager.cc:1905] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:34:00.249517 18234 tablet_bootstrap.cc:492] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:34:00.250134 18234 ts_tablet_manager.cc:1397] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.007s
I20260810 23:34:00.250696 17087 tablet_service.cc:1514] Processing DeleteTablet for tablet 783e71cfbc904ffc9d19e5b9257709c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d09b4ee48905498991a54e4c98e9381d) from {username='slave'} at 127.0.0.1:53976
W20260810 23:34:00.252851 17716 catalog_manager.cc:4821] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): delete failed for tablet 783e71cfbc904ffc9d19e5b9257709c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 783e71cfbc904ffc9d19e5b9257709c5 already in progress: creating tablet
I20260810 23:34:00.254791 18234 raft_consensus.cc:348] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.255698 18234 raft_consensus.cc:374] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:00.256011 18234 raft_consensus.cc:729] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:34:00.256784 17304 tablet_service.cc:1467] Processing CreateTablet for tablet d09b4ee48905498991a54e4c98e9381d (DEFAULT_TABLE table=default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:34:00.258076 17304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09b4ee48905498991a54e4c98e9381d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:34:00.266675 17731 master_service.cc:946] ReplaceTablet: received request to replace tablet 783e71cfbc904ffc9d19e5b9257709c5 from {username='slave'} at 127.0.0.1:48402
I20260810 23:34:00.266134 18234 consensus_queue.cc:260] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.276487 18234 ts_tablet_manager.cc:1428] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.026s user 0.010s sys 0.000s
I20260810 23:34:00.288585 18234 tablet_bootstrap.cc:492] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:34:00.293045 18221 ts_tablet_manager.cc:1918] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:34:00.293581 18221 log.cc:1196] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/783e71cfbc904ffc9d19e5b9257709c5
I20260810 23:34:00.294371 18221 ts_tablet_manager.cc:1939] T 783e71cfbc904ffc9d19e5b9257709c5 P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:34:00.297403 17087 tablet_service.cc:1514] Processing DeleteTablet for tablet 783e71cfbc904ffc9d19e5b9257709c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d09b4ee48905498991a54e4c98e9381d) from {username='slave'} at 127.0.0.1:53976
I20260810 23:34:00.298282 18227 tablet_replica.cc:330] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:34:00.299065 18227 raft_consensus.cc:2229] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:34:00.299557 18227 raft_consensus.cc:2258] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:00.302416 18227 ts_tablet_manager.cc:1905] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:34:00.303333 17720 catalog_manager.cc:4841] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): tablet 783e71cfbc904ffc9d19e5b9257709c5 (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]) successfully deleted
I20260810 23:34:00.307456 17745 master_service.cc:946] ReplaceTablet: received request to replace tablet f35cb671e0ea436b9f155d4cc7f31bd8 from {username='slave'} at 127.0.0.1:48402
I20260810 23:34:00.311172 18247 raft_consensus.cc:482] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:00.311673 18247 raft_consensus.cc:504] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.313912 18247 leader_election.cc:288] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:34:00.315900 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35cb671e0ea436b9f155d4cc7f31bd8" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2466b33eeb034688bcf670ff612d452a" is_pre_election: true
I20260810 23:34:00.317072 17341 raft_consensus.cc:2454] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a651570d6e459d98bcf97054e07921 in term 0.
I20260810 23:34:00.319340 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35cb671e0ea436b9f155d4cc7f31bd8" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36" is_pre_election: true
I20260810 23:34:00.319396 17030 leader_election.cc:302] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a, 80a651570d6e459d98bcf97054e07921; no voters:
I20260810 23:34:00.320310 16904 raft_consensus.cc:2454] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a651570d6e459d98bcf97054e07921 in term 0.
I20260810 23:34:00.320755 18247 raft_consensus.cc:2781] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:34:00.321185 18247 raft_consensus.cc:482] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:34:00.321640 18247 raft_consensus.cc:3037] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:34:00.328428 18234 tablet_bootstrap.cc:654] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:00.344375 17745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Sending DeleteTablet for 3 replicas of tablet f35cb671e0ea436b9f155d4cc7f31bd8
I20260810 23:34:00.333492 18247 raft_consensus.cc:504] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.346702 16855 tablet_service.cc:1514] Processing DeleteTablet for tablet f35cb671e0ea436b9f155d4cc7f31bd8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b) from {username='slave'} at 127.0.0.1:36456
I20260810 23:34:00.347463 18218 tablet_bootstrap.cc:492] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:34:00.349550 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35cb671e0ea436b9f155d4cc7f31bd8" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36"
I20260810 23:34:00.350665 17303 tablet_service.cc:1514] Processing DeleteTablet for tablet f35cb671e0ea436b9f155d4cc7f31bd8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b) from {username='slave'} at 127.0.0.1:57860
I20260810 23:34:00.351482 18228 tablet_replica.cc:330] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:34:00.347568 18221 tablet_replica.cc:330] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:34:00.352301 18228 raft_consensus.cc:2229] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:34:00.352912 18228 raft_consensus.cc:2258] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:00.355001 17091 tablet_service.cc:1514] Processing DeleteTablet for tablet f35cb671e0ea436b9f155d4cc7f31bd8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b) from {username='slave'} at 127.0.0.1:53976
I20260810 23:34:00.355096 16904 raft_consensus.cc:3037] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:34:00.355680 18228 ts_tablet_manager.cc:1905] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:34:00.357493 18247 leader_election.cc:288] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 election: Requested vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949)
I20260810 23:34:00.357791 18218 tablet_bootstrap.cc:654] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:00.358323 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f35cb671e0ea436b9f155d4cc7f31bd8" candidate_uuid: "80a651570d6e459d98bcf97054e07921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2466b33eeb034688bcf670ff612d452a"
I20260810 23:34:00.359962 18220 tablet_bootstrap.cc:492] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
W20260810 23:34:00.361130 17030 leader_election.cc:341] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): Illegal state: Tablet not RUNNING: STOPPED
I20260810 23:34:00.367060 18220 tablet_bootstrap.cc:654] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:00.370894 17745 catalog_manager.cc:6565] ReplaceTablet: tablet f35cb671e0ea436b9f155d4cc7f31bd8 deleted and replaced by tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b
I20260810 23:34:00.385659 18234 tablet_bootstrap.cc:492] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:34:00.386247 18234 ts_tablet_manager.cc:1397] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.098s user 0.002s sys 0.009s
I20260810 23:34:00.388491 16904 raft_consensus.cc:2454] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a651570d6e459d98bcf97054e07921 in term 1.
I20260810 23:34:00.389351 18221 raft_consensus.cc:2229] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:34:00.390658 17031 leader_election.cc:302] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36; no voters: 2466b33eeb034688bcf670ff612d452a
I20260810 23:34:00.391377 18221 raft_consensus.cc:2258] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:00.392118 18247 raft_consensus.cc:2781] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:34:00.393574 18221 ts_tablet_manager.cc:1905] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:34:00.397444 18234 raft_consensus.cc:348] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.398340 18234 raft_consensus.cc:374] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:00.398628 18234 raft_consensus.cc:729] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:34:00.398656 16860 tablet_service.cc:1467] Processing CreateTablet for tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:34:00.400162 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:34:00.406358 18220 tablet_bootstrap.cc:492] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:34:00.406888 18220 ts_tablet_manager.cc:1397] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.005s
I20260810 23:34:00.410012 17091 tablet_service.cc:1467] Processing CreateTablet for tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:34:00.411463 17091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:34:00.418391 17304 tablet_service.cc:1467] Processing CreateTablet for tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:34:00.420010 17304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:34:00.422243 18218 tablet_bootstrap.cc:492] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:34:00.422762 18218 ts_tablet_manager.cc:1397] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.076s user 0.013s sys 0.003s
I20260810 23:34:00.425714 18218 raft_consensus.cc:348] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
@ 0x7f6e81e73aea kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260810 23:34:00.433283 18247 raft_consensus.cc:686] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 1 LEADER]: Becoming Leader. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Running, Role: LEADER
I20260810 23:34:00.434406 18247 consensus_queue.cc:237] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.426522 18218 raft_consensus.cc:374] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:00.435906 18218 raft_consensus.cc:729] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:34:00.436780 18218 consensus_queue.cc:260] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.438819 18220 raft_consensus.cc:348] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.442462 18228 ts_tablet_manager.cc:1918] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:34:00.442798 18228 log.cc:1196] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/f35cb671e0ea436b9f155d4cc7f31bd8
I20260810 23:34:00.443427 18228 ts_tablet_manager.cc:1939] T f35cb671e0ea436b9f155d4cc7f31bd8 P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:34:00.399403 18234 consensus_queue.cc:260] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.446368 18234 ts_tablet_manager.cc:1428] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.060s user 0.008s sys 0.000s
I20260810 23:34:00.455088 18218 ts_tablet_manager.cc:1428] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.032s user 0.008s sys 0.001s
I20260810 23:34:00.457382 18227 ts_tablet_manager.cc:1918] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:34:00.457950 18227 log.cc:1196] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/783e71cfbc904ffc9d19e5b9257709c5
I20260810 23:34:00.459025 18227 ts_tablet_manager.cc:1939] T 783e71cfbc904ffc9d19e5b9257709c5 P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:34:00.459594 18218 tablet_bootstrap.cc:492] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:34:00.439952 18220 raft_consensus.cc:374] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:00.461797 18220 raft_consensus.cc:729] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:34:00.463218 18220 consensus_queue.cc:260] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.474977 18220 ts_tablet_manager.cc:1428] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.068s user 0.009s sys 0.000s
I20260810 23:34:00.473135 18221 ts_tablet_manager.cc:1918] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:34:00.479424 18221 log.cc:1196] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/f35cb671e0ea436b9f155d4cc7f31bd8
I20260810 23:34:00.480280 18221 ts_tablet_manager.cc:1939] T f35cb671e0ea436b9f155d4cc7f31bd8 P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:34:00.482630 17716 catalog_manager.cc:4841] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): tablet 783e71cfbc904ffc9d19e5b9257709c5 (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]) successfully deleted
I20260810 23:34:00.483546 17731 master_service.cc:946] ReplaceTablet: received request to replace tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b from {username='slave'} at 127.0.0.1:48402
I20260810 23:34:00.492672 18218 tablet_bootstrap.cc:654] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:00.494407 18227 tablet_replica.cc:330] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:34:00.495949 18227 raft_consensus.cc:2229] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:34:00.496794 18227 pending_rounds.cc:62] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921: Trying to abort 1 pending ops.
I20260810 23:34:00.497027 18227 pending_rounds.cc:68] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317114226430369792 op_type: NO_OP noop_request { }
I20260810 23:34:00.497491 18227 raft_consensus.cc:2866] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 23:34:00.497853 18227 raft_consensus.cc:2258] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:00.501302 18227 ts_tablet_manager.cc:1905] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:34:00.505580 18234 tablet_bootstrap.cc:492] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:34:00.512153 18234 tablet_bootstrap.cc:654] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:00.520501 18220 tablet_bootstrap.cc:492] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:34:00.526996 18218 tablet_bootstrap.cc:492] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:34:00.527570 18218 ts_tablet_manager.cc:1397] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.008s
I20260810 23:34:00.528563 18220 tablet_bootstrap.cc:654] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:00.529229 18234 tablet_bootstrap.cc:492] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:34:00.529840 18234 ts_tablet_manager.cc:1397] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.004s
I20260810 23:34:00.530678 18218 raft_consensus.cc:348] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.531554 18218 raft_consensus.cc:374] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:00.533305 18234 raft_consensus.cc:348] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.534037 18234 raft_consensus.cc:374] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:00.534279 18234 raft_consensus.cc:729] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
@ 0x7f6e81e7333c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260810 23:34:00.537330 17720 catalog_manager.cc:4841] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): tablet f35cb671e0ea436b9f155d4cc7f31bd8 (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
I20260810 23:34:00.531873 18218 raft_consensus.cc:729] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:34:00.544652 18234 consensus_queue.cc:260] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.546000 18218 consensus_queue.cc:260] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.549513 18218 ts_tablet_manager.cc:1428] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.022s user 0.003s sys 0.006s
I20260810 23:34:00.552279 18234 ts_tablet_manager.cc:1428] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.022s user 0.010s sys 0.001s
I20260810 23:34:00.559484 18227 ts_tablet_manager.cc:1918] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 23:34:00.560470 18227 log.cc:1196] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/f35cb671e0ea436b9f155d4cc7f31bd8
I20260810 23:34:00.560806 17731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Sending DeleteTablet for 3 replicas of tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b
I20260810 23:34:00.561725 18227 ts_tablet_manager.cc:1939] T f35cb671e0ea436b9f155d4cc7f31bd8 P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:34:00.566797 17716 catalog_manager.cc:4841] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): tablet f35cb671e0ea436b9f155d4cc7f31bd8 (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
@ 0x7f6e81e76fd2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260810 23:34:00.581970 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b with delete_type TABLET_DATA_DELETED (replaced by tablet ae48313393b146eca3717b568666bd0e) from {username='slave'} at 127.0.0.1:57860
I20260810 23:34:00.583847 17091 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b with delete_type TABLET_DATA_DELETED (replaced by tablet ae48313393b146eca3717b568666bd0e) from {username='slave'} at 127.0.0.1:53976
I20260810 23:34:00.584939 18227 tablet_replica.cc:330] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:34:00.585804 18227 raft_consensus.cc:2229] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:34:00.586006 17731 catalog_manager.cc:6565] ReplaceTablet: tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b deleted and replaced by tablet ae48313393b146eca3717b568666bd0e
I20260810 23:34:00.586153 16860 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b with delete_type TABLET_DATA_DELETED (replaced by tablet ae48313393b146eca3717b568666bd0e) from {username='slave'} at 127.0.0.1:36456
I20260810 23:34:00.586521 18227 raft_consensus.cc:2258] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:00.587121 18221 tablet_replica.cc:330] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:34:00.588186 18221 raft_consensus.cc:2229] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:34:00.588752 18221 raft_consensus.cc:2258] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:00.589466 18227 ts_tablet_manager.cc:1905] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:34:00.594242 18221 ts_tablet_manager.cc:1905] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:34:00.625926 17729 master_service.cc:946] ReplaceTablet: received request to replace tablet d09b4ee48905498991a54e4c98e9381d from {username='slave'} at 127.0.0.1:48402
I20260810 23:34:00.634732 18227 ts_tablet_manager.cc:1918] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:34:00.635296 18227 log.cc:1196] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/0aac0f7bd40849c3a9e2c1d54ce0a90b
I20260810 23:34:00.636276 18227 ts_tablet_manager.cc:1939] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:34:00.643002 17718 catalog_manager.cc:4841] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): tablet f35cb671e0ea436b9f155d4cc7f31bd8 (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
I20260810 23:34:00.645076 18221 ts_tablet_manager.cc:1918] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:34:00.655211 17716 catalog_manager.cc:4841] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
W20260810 23:34:00.655419 17718 catalog_manager.cc:4821] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): delete failed for tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b already in progress: creating tablet
I20260810 23:34:00.645557 18221 log.cc:1196] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/0aac0f7bd40849c3a9e2c1d54ce0a90b
I20260810 23:34:00.657191 18221 ts_tablet_manager.cc:1939] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:34:00.660279 17720 catalog_manager.cc:4841] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b (table default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]) successfully deleted
I20260810 23:34:00.660871 18220 tablet_bootstrap.cc:492] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:34:00.661330 18220 ts_tablet_manager.cc:1397] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.141s user 0.059s sys 0.057s
I20260810 23:34:00.664929 18220 raft_consensus.cc:348] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.669114 17729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0596cdb0a514cc08d7df0ceea528339: Sending DeleteTablet for 3 replicas of tablet d09b4ee48905498991a54e4c98e9381d
I20260810 23:34:00.670758 17729 catalog_manager.cc:6565] ReplaceTablet: tablet d09b4ee48905498991a54e4c98e9381d deleted and replaced by tablet ac2a8def0d7d4e00a330097bd76f6ba3
@ 0x7f6e81e76f8a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260810 23:34:00.672267 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet d09b4ee48905498991a54e4c98e9381d with delete_type TABLET_DATA_DELETED (replaced by tablet ac2a8def0d7d4e00a330097bd76f6ba3) from {username='slave'} at 127.0.0.1:57860
I20260810 23:34:00.673286 18228 tablet_replica.cc:330] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:34:00.673280 17091 tablet_service.cc:1514] Processing DeleteTablet for tablet d09b4ee48905498991a54e4c98e9381d with delete_type TABLET_DATA_DELETED (replaced by tablet ac2a8def0d7d4e00a330097bd76f6ba3) from {username='slave'} at 127.0.0.1:53976
I20260810 23:34:00.674190 18227 tablet_replica.cc:330] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921: stopping tablet replica
I20260810 23:34:00.674165 18228 raft_consensus.cc:2229] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:34:00.674600 16860 tablet_service.cc:1514] Processing DeleteTablet for tablet d09b4ee48905498991a54e4c98e9381d with delete_type TABLET_DATA_DELETED (replaced by tablet ac2a8def0d7d4e00a330097bd76f6ba3) from {username='slave'} at 127.0.0.1:36456
I20260810 23:34:00.674695 18228 raft_consensus.cc:2258] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:00.674973 18227 raft_consensus.cc:2229] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:34:00.675495 18227 raft_consensus.cc:2258] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:00.665623 18220 raft_consensus.cc:374] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:00.675885 18220 raft_consensus.cc:729] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:34:00.676697 18220 consensus_queue.cc:260] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.677237 18228 ts_tablet_manager.cc:1905] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:34:00.679046 18220 ts_tablet_manager.cc:1428] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260810 23:34:00.679024 17091 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b with delete_type TABLET_DATA_DELETED (replaced by tablet ae48313393b146eca3717b568666bd0e) from {username='slave'} at 127.0.0.1:53976
I20260810 23:34:00.678081 18227 ts_tablet_manager.cc:1905] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f6e81e76f1a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260810 23:34:00.704876 18221 tablet_replica.cc:330] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36: stopping tablet replica
I20260810 23:34:00.705953 18221 raft_consensus.cc:2229] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:34:00.706594 18221 raft_consensus.cc:2258] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:00.717125 16860 tablet_service.cc:1467] Processing CreateTablet for tablet ae48313393b146eca3717b568666bd0e (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:34:00.718525 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae48313393b146eca3717b568666bd0e. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f6e81e76ee2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260810 23:34:00.727679 17304 tablet_service.cc:1467] Processing CreateTablet for tablet ae48313393b146eca3717b568666bd0e (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:34:00.727739 17091 tablet_service.cc:1467] Processing CreateTablet for tablet ae48313393b146eca3717b568666bd0e (DEFAULT_TABLE table=default.table_edb188d6a21846618b6fc3fe0927de7c [id=54d3236abec942629222f30460ddaa97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:34:00.729125 17304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae48313393b146eca3717b568666bd0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:34:00.729240 17091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae48313393b146eca3717b568666bd0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:34:00.738313 17303 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b with delete_type TABLET_DATA_DELETED (replaced by tablet ae48313393b146eca3717b568666bd0e) from {username='slave'} at 127.0.0.1:57860
I20260810 23:34:00.744900 18221 ts_tablet_manager.cc:1905] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f6e81e761de std::__1::__function::__func<>::operator()() at ??:0
@ 0x580785 std::__1::__function::__value_func<>::operator()() at ??:?
I20260810 23:34:00.750312 18227 ts_tablet_manager.cc:1918] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:34:00.750819 18227 log.cc:1196] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-1/wal/wals/d09b4ee48905498991a54e4c98e9381d
@ 0x5805ba std::__1::function<>::operator()() at ??:?
I20260810 23:34:00.751602 18227 ts_tablet_manager.cc:1939] T d09b4ee48905498991a54e4c98e9381d P 80a651570d6e459d98bcf97054e07921: Deleting consensus metadata
I20260810 23:34:00.754853 17716 catalog_manager.cc:4841] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): tablet d09b4ee48905498991a54e4c98e9381d (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]) successfully deleted
W20260810 23:34:00.762382 17716 catalog_manager.cc:4808] TS 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): delete failed for tablet 0aac0f7bd40849c3a9e2c1d54ce0a90b because the tablet was not found. No further retry: Not found: Tablet not found: 0aac0f7bd40849c3a9e2c1d54ce0a90b
I20260810 23:34:00.765879 18218 tablet_bootstrap.cc:492] T ae48313393b146eca3717b568666bd0e P c3b587fa042b42e4b66bb113245c0e36: Bootstrap starting.
I20260810 23:34:00.773257 18218 tablet_bootstrap.cc:654] T ae48313393b146eca3717b568666bd0e P c3b587fa042b42e4b66bb113245c0e36: Neither blocks nor log segments found. Creating new log.
@ 0x7f6e78f89dc7 kudu::Thread::SuperviseThread() at ??:0
@ 0x4230f0 __tsan_thread_start_func at ??:?
I20260810 23:34:00.783595 18221 ts_tablet_manager.cc:1918] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:34:00.785174 18234 tablet_bootstrap.cc:492] T ae48313393b146eca3717b568666bd0e P 80a651570d6e459d98bcf97054e07921: Bootstrap starting.
I20260810 23:34:00.789055 18221 log.cc:1196] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-0/wal/wals/d09b4ee48905498991a54e4c98e9381d
I20260810 23:34:00.789906 18221 ts_tablet_manager.cc:1939] T d09b4ee48905498991a54e4c98e9381d P c3b587fa042b42e4b66bb113245c0e36: Deleting consensus metadata
I20260810 23:34:00.792537 18234 tablet_bootstrap.cc:654] T ae48313393b146eca3717b568666bd0e P 80a651570d6e459d98bcf97054e07921: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:00.793263 17720 catalog_manager.cc:4841] TS c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): tablet d09b4ee48905498991a54e4c98e9381d (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]) successfully deleted
I20260810 23:34:00.796701 18218 tablet_bootstrap.cc:492] T ae48313393b146eca3717b568666bd0e P c3b587fa042b42e4b66bb113245c0e36: No bootstrap required, opened a new log
I20260810 23:34:00.797211 18218 ts_tablet_manager.cc:1397] T ae48313393b146eca3717b568666bd0e P c3b587fa042b42e4b66bb113245c0e36: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.005s
I20260810 23:34:00.800818 17731 master_service.cc:946] ReplaceTablet: received request to replace tablet ae48313393b146eca3717b568666bd0e from {username='slave'} at 127.0.0.1:48402
@ 0x7f6e7a5f76db start_thread at ??:0
I20260810 23:34:00.804404 18218 raft_consensus.cc:348] T ae48313393b146eca3717b568666bd0e P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.805465 18218 raft_consensus.cc:374] T ae48313393b146eca3717b568666bd0e P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:00.806134 18218 raft_consensus.cc:729] T ae48313393b146eca3717b568666bd0e P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b587fa042b42e4b66bb113245c0e36, State: Initialized, Role: FOLLOWER
I20260810 23:34:00.807019 18218 consensus_queue.cc:260] T ae48313393b146eca3717b568666bd0e P c3b587fa042b42e4b66bb113245c0e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.813076 18234 tablet_bootstrap.cc:492] T ae48313393b146eca3717b568666bd0e P 80a651570d6e459d98bcf97054e07921: No bootstrap required, opened a new log
I20260810 23:34:00.813619 18234 ts_tablet_manager.cc:1397] T ae48313393b146eca3717b568666bd0e P 80a651570d6e459d98bcf97054e07921: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.001s
I20260810 23:34:00.816895 18218 ts_tablet_manager.cc:1428] T ae48313393b146eca3717b568666bd0e P c3b587fa042b42e4b66bb113245c0e36: Time spent starting tablet: real 0.019s user 0.006s sys 0.002s
I20260810 23:34:00.816772 18234 raft_consensus.cc:348] T ae48313393b146eca3717b568666bd0e P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.817569 18234 raft_consensus.cc:374] T ae48313393b146eca3717b568666bd0e P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:00.817857 18234 raft_consensus.cc:729] T ae48313393b146eca3717b568666bd0e P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a651570d6e459d98bcf97054e07921, State: Initialized, Role: FOLLOWER
I20260810 23:34:00.818609 18234 consensus_queue.cc:260] T ae48313393b146eca3717b568666bd0e P 80a651570d6e459d98bcf97054e07921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.826442 18220 tablet_bootstrap.cc:492] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a: Bootstrap starting.
I20260810 23:34:00.830103 18228 ts_tablet_manager.cc:1918] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:34:00.830669 18228 log.cc:1196] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/d09b4ee48905498991a54e4c98e9381d
I20260810 23:34:00.831559 18228 ts_tablet_manager.cc:1939] T d09b4ee48905498991a54e4c98e9381d P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:34:00.832825 18220 tablet_bootstrap.cc:654] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:00.839797 18234 ts_tablet_manager.cc:1428] T ae48313393b146eca3717b568666bd0e P 80a651570d6e459d98bcf97054e07921: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260810 23:34:00.847322 18228 tablet_replica.cc:330] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a: stopping tablet replica
I20260810 23:34:00.848279 18228 raft_consensus.cc:2229] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 23:34:00.848402 17718 catalog_manager.cc:4841] TS 2466b33eeb034688bcf670ff612d452a (127.11.226.131:39949): tablet d09b4ee48905498991a54e4c98e9381d (table default.table_edccecac3c7e4be4a5e3eab151d78907 [id=2a70ac3899e649deb7fbd6e3bbf1d43d]) successfully deleted
I20260810 23:34:00.848829 18228 raft_consensus.cc:2258] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Raft consensus is shut down!
@ 0x7f6e75a7d61f clone at ??:0
I20260810 23:34:00.855170 18220 tablet_bootstrap.cc:492] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a: No bootstrap required, opened a new log
I20260810 23:34:00.855715 18220 ts_tablet_manager.cc:1397] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.000s
I20260810 23:34:00.860191 18220 raft_consensus.cc:348] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.860960 18220 raft_consensus.cc:374] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:00.861276 18220 raft_consensus.cc:729] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Initialized, Role: FOLLOWER
I20260810 23:34:00.861752 18228 ts_tablet_manager.cc:1905] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:34:00.862071 18220 consensus_queue.cc:260] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.864610 18220 ts_tablet_manager.cc:1428] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260810 23:34:00.909914 18240 raft_consensus.cc:482] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:00.910490 18240 raft_consensus.cc:504] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.912743 18240 leader_election.cc:288] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
I20260810 23:34:00.913964 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae48313393b146eca3717b568666bd0e" candidate_uuid: "2466b33eeb034688bcf670ff612d452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36" is_pre_election: true
I20260810 23:34:00.914722 16904 raft_consensus.cc:2454] T ae48313393b146eca3717b568666bd0e P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2466b33eeb034688bcf670ff612d452a in term 0.
I20260810 23:34:00.915222 17126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae48313393b146eca3717b568666bd0e" candidate_uuid: "2466b33eeb034688bcf670ff612d452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a651570d6e459d98bcf97054e07921" is_pre_election: true
I20260810 23:34:00.916076 17126 raft_consensus.cc:2454] T ae48313393b146eca3717b568666bd0e P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2466b33eeb034688bcf670ff612d452a in term 0.
I20260810 23:34:00.916242 17237 leader_election.cc:302] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a, c3b587fa042b42e4b66bb113245c0e36; no voters:
I20260810 23:34:00.917243 18240 raft_consensus.cc:2781] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:34:00.917596 18240 raft_consensus.cc:482] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:34:00.917900 18240 raft_consensus.cc:3037] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:34:00.926822 18228 ts_tablet_manager.cc:1918] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 23:34:00.926623 18240 raft_consensus.cc:504] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:00.927482 18228 log.cc:1196] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a: Deleting WAL directory at /tmp/dist-test-taskt0RZpf/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1786404722205025-12170-0/minicluster-data/ts-2/wal/wals/0aac0f7bd40849c3a9e2c1d54ce0a90b
I20260810 23:34:00.930610 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae48313393b146eca3717b568666bd0e" candidate_uuid: "2466b33eeb034688bcf670ff612d452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b587fa042b42e4b66bb113245c0e36"
I20260810 23:34:00.931653 16904 raft_consensus.cc:3037] T ae48313393b146eca3717b568666bd0e P c3b587fa042b42e4b66bb113245c0e36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:34:00.934819 18228 ts_tablet_manager.cc:1939] T 0aac0f7bd40849c3a9e2c1d54ce0a90b P 2466b33eeb034688bcf670ff612d452a: Deleting consensus metadata
I20260810 23:34:00.940364 16904 raft_consensus.cc:2454] T ae48313393b146eca3717b568666bd0e P c3b587fa042b42e4b66bb113245c0e36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2466b33eeb034688bcf670ff612d452a in term 1.
I20260810 23:34:00.942968 17126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae48313393b146eca3717b568666bd0e" candidate_uuid: "2466b33eeb034688bcf670ff612d452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a651570d6e459d98bcf97054e07921"
I20260810 23:34:00.943642 17126 raft_consensus.cc:3037] T ae48313393b146eca3717b568666bd0e P 80a651570d6e459d98bcf97054e07921 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:34:00.952764 18240 leader_election.cc:288] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 election: Requested vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
I20260810 23:34:00.953718 17126 raft_consensus.cc:2454] T ae48313393b146eca3717b568666bd0e P 80a651570d6e459d98bcf97054e07921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2466b33eeb034688bcf670ff612d452a in term 1.
I20260810 23:34:00.955265 17237 leader_election.cc:302] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a, c3b587fa042b42e4b66bb113245c0e36; no voters:
I20260810 23:34:00.961429 18286 raft_consensus.cc:2781] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:34:00.963176 18286 raft_consensus.cc:686] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [term 1 LEADER]: Becoming Leader. State: Replica: 2466b33eeb034688bcf670ff612d452a, State: Running, Role: LEADER
I20260810 23:34:00.964120 18286 consensus_queue.cc:237] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:01.178927 18286 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:01.179562 18286 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:01.182082 18286 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:34:01.184265 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:34:01.184815 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:34:01.185266 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:34:01.186053 18286 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:34:01.186942 17237 connection.cc:535] server connection from 127.0.0.1:57860 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:34:01.187672 17236 connection.cc:535] client connection to 127.11.226.189:34939 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:34:01.194271 18179 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:34:01.192509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.226.189:34939 (local address 127.11.226.129:57193)
0810 23:34:01.193132 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0810 23:34:01.193375 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":110}
W20260810 23:34:01.195070 17844 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:34:01.190335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.226.189:34939 (local address 127.11.226.130:56839)
0810 23:34:01.191008 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0810 23:34:01.191060 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:34:01.191377 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:34:01.194115 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":130}
W20260810 23:34:01.195199 17428 connection.cc:535] client connection to 127.11.226.189:34939 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:34:01.197243 17428 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.226.189:34939)
W20260810 23:34:01.198626 17428 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.226.189:34939)
W20260810 23:34:01.199334 17428 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.226.189:34939)
W20260810 23:34:01.200977 17428 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.226.189:34939)
W20260810 23:34:01.201704 17992 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:34:01.191212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.226.189:34939 (local address 127.11.226.131:32773)
0810 23:34:01.192110 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0810 23:34:01.192172 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:34:01.192424 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:34:01.200738 (+ 8314us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":347}
I20260810 23:34:01.338258 18291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:01.338950 18291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:01.342407 18291 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:34:01.349200 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:01.361016 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:01.361758 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:01.362116 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:01.363130 18291 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:34:01.545181 12170 master-stress-test.cc:578] Killing test thread 1/10
I20260810 23:34:01.935078 18298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:01.935032 18297 consensus_queue.cc:1035] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 23:34:01.935708 18298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:01.948447 18298 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
I20260810 23:34:02.000679 18307 consensus_queue.cc:1035] T ae48313393b146eca3717b568666bd0e P 2466b33eeb034688bcf670ff612d452a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20260810 23:34:02.030593 18056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.226.190:42189, user_credentials={real_user=slave}} blocked reactor thread for 62787.1us
W20260810 23:34:02.035112 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:02.043527 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:02.044204 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:02.045104 18298 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:34:02.757109 18339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:02.757643 18339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:02.759538 18339 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:34:02.766335 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:02.767006 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:02.767392 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:02.768222 18339 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:34:03.832758 18363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:03.833396 18363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:03.835219 18363 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:34:03.840960 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:03.841697 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:03.842010 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:03.842624 18363 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:34:04.239152 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:34:04.239645 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:34:04.629904 18383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:04.630410 18383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:04.632117 18383 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:34:04.637034 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:04.637614 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:04.637953 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:04.638562 18383 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:34:06.954205 18433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:06.954691 18433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:06.956619 18433 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:34:06.959697 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 23:34:06.963122 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:06.963663 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:06.963984 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:06.964682 18433 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:34:07.982775 18455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:07.983711 18455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:07.988466 18455 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:34:08.014436 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:08.015187 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:08.015604 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:08.016544 18455 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 23:34:08.979575 17400 debug-util.cc:398] Leaking SignalData structure 0x7b08000da080 after lost signal to thread 17207
W20260810 23:34:08.981020 17400 debug-util.cc:398] Leaking SignalData structure 0x7b0800042860 after lost signal to thread 17405
I20260810 23:34:09.249761 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:34:09.250522 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:34:09.704123 18511 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:09.704996 18511 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:34:09.708521 18511 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:34:09.711452 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:34:09.712525 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:34:09.712903 17237 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:34:09.713689 18511 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:34:11.303297 18536 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:11.303877 18536 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:11.306074 18536 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:34:11.308295 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:34:11.308852 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:34:11.309235 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:34:11.309962 18536 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:34:13.247502 18561 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:13.248293 18561 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:34:13.251040 18561 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:34:13.253571 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:34:13.254123 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:34:13.254542 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:34:13.255414 18561 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:34:14.254390 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:34:14.254906 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:34:15.291431 18629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:15.292069 18629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:15.293886 18629 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:34:15.297693 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260810 23:34:15.301597 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:15.302296 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:15.302656 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:15.303583 18629 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:34:19.259788 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:34:19.260587 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:34:19.356376 18722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:19.357095 18722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:19.360175 18722 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:34:19.365706 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:19.366413 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:19.366709 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:19.367533 18722 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:34:21.223393 18763 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:21.224443 18763 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:21.227545 18763 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:34:21.229832 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:34:21.230676 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:34:21.231066 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:34:21.231830 18763 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:34:24.264652 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:34:24.265202 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:34:27.089016 18885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:27.089589 18885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:27.091640 18885 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:34:27.094391 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260810 23:34:27.097332 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:27.097981 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:27.098229 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:27.098800 18885 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:34:29.269857 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:34:29.270601 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:34:31.564596 18981 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:31.565327 18981 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:34:31.568353 18981 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:34:31.570930 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:34:31.571672 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:34:31.572113 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:34:31.572897 18981 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:34:31.970885 18991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:31.971735 18991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:31.974642 18991 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:34:31.981103 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:31.981875 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:31.982172 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:31.982929 18991 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:34:34.274243 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:34:34.274744 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:34:39.279162 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:34:39.279711 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:34:39.445299 19152 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:39.446161 19152 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:39.448659 19152 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:34:39.450908 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:34:39.451601 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:34:39.452006 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:34:39.452839 19152 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:34:40.033284 17189 debug-util.cc:398] Leaking SignalData structure 0x7b08000af900 after lost signal to thread 16999
W20260810 23:34:40.034688 17189 debug-util.cc:398] Leaking SignalData structure 0x7b08000c38c0 after lost signal to thread 17192
I20260810 23:34:42.381763 19194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:42.382386 19194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:42.384820 19194 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:34:42.387197 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260810 23:34:42.391844 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:42.392556 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:42.392930 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:42.393728 19194 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:34:44.283143 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:34:44.283650 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:34:45.889954 19270 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:45.890559 19270 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:34:45.892511 19270 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:34:45.894606 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:34:45.895233 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:34:45.895677 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:34:45.896418 19270 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:34:46.267849 19270 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:46.268532 19270 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:34:46.270752 19270 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:34:46.273092 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:34:46.273579 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:34:46.273985 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:34:46.274688 19270 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:34:47.738667 19312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:47.739598 19312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:47.742427 19312 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:34:47.746308 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:34:47.750336 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:47.750950 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:47.751211 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:47.752066 19312 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 23:34:48.464177 16979 heartbeater.cc:644] Failed to heartbeat to 127.11.226.189:34939 (96 consecutive failures): Network error: Failed to ping master at 127.11.226.189:34939: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
W20260810 23:34:48.628674 17195 heartbeater.cc:644] Failed to heartbeat to 127.11.226.190:42189 (128 consecutive failures): Network error: Failed to ping master at 127.11.226.190:42189: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:48.887877 17409 heartbeater.cc:644] Failed to heartbeat to 127.11.226.189:34939 (97 consecutive failures): Network error: Failed to ping master at 127.11.226.189:34939: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:49.287889 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:34:49.288542 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:34:54.292172 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:34:54.292775 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:34:55.288836 19463 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:55.289553 19463 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:34:55.292232 19463 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:34:55.294951 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:34:55.295557 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:34:55.295949 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:34:55.296782 19463 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:34:57.480839 19504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:34:57.481647 19504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:34:57.484521 19504 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:34:57.488113 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:34:57.492246 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:34:57.492882 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:34:57.493202 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:34:57.493957 19504 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:34:59.296967 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:34:59.297571 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:35:03.398579 19626 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:03.399537 19626 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:35:03.402264 19626 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:35:03.404613 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:35:03.405253 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:35:03.405682 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:35:03.406498 19626 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:35:04.301219 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:35:04.301748 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:35:05.446609 19674 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:05.447338 19674 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:35:05.449513 19674 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:35:05.451630 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:35:05.452252 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:35:05.452603 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:35:05.453392 19674 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:35:06.576364 19693 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:06.577387 19693 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:35:06.580431 19693 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:35:06.582823 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:35:06.583731 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:35:06.584267 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:35:06.585271 19693 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:35:09.306392 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:35:09.307085 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:35:13.505784 19862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:35:13.506450 19862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:35:13.509362 19862 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:35:13.512167 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:35:13.516688 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:35:13.517369 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:35:13.517683 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:35:13.518427 19862 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:35:14.310751 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:35:14.311369 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:35:15.816104 19923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:35:15.816663 19923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:35:15.818511 19923 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:35:15.824891 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:35:15.825486 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:35:15.825801 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:35:15.826442 19923 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:35:19.315762 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:35:19.316493 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:35:21.250141 20031 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:21.251021 20031 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:35:21.254987 20031 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:35:21.257191 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:35:21.257954 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:35:21.258352 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:35:21.259212 20031 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:35:21.833655 20049 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:21.834147 20049 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:35:21.836131 20049 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:35:21.838106 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:35:21.838620 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:35:21.838972 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:35:21.839622 20049 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:35:23.215211 20071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:35:23.216179 20071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:35:23.218343 20071 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:35:23.222111 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260810 23:35:23.225397 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:35:23.226058 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:35:23.226403 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:35:23.227207 20071 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 23:35:23.456907 16972 debug-util.cc:398] Leaking SignalData structure 0x7b08000d46c0 after lost signal to thread 16737
W20260810 23:35:23.458277 16972 debug-util.cc:398] Leaking SignalData structure 0x7b0800043080 after lost signal to thread 16976
I20260810 23:35:24.320418 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:35:24.320981 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:35:25.969919 20129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:35:25.970498 20129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:35:25.972913 20129 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:35:25.980280 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:35:25.980901 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:35:25.981230 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:35:25.982398 20129 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:35:29.325639 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:35:29.326170 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:35:29.877343 17189 debug-util.cc:398] Leaking SignalData structure 0x7b08000dddc0 after lost signal to thread 16999
W20260810 23:35:29.878552 17189 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8680 after lost signal to thread 17192
I20260810 23:35:30.510135 20231 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:30.510834 20231 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:35:30.512939 20231 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:35:30.515743 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:35:30.516417 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:35:30.516716 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:35:30.517510 20231 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 23:35:32.431934 18140 debug-util.cc:398] Leaking SignalData structure 0x7b08000502e0 after lost signal to thread 17993
W20260810 23:35:32.432828 18140 debug-util.cc:398] Leaking SignalData structure 0x7b080005b4a0 after lost signal to thread 18143
I20260810 23:35:34.381330 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:35:34.381943 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:35:36.702870 20363 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:36.703786 20363 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:35:36.707100 20363 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:35:36.709422 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:35:36.710067 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:35:36.710434 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:35:36.711170 20363 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:35:39.386004 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:35:39.386530 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:35:40.082551 20444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:35:40.083163 20444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:35:40.085073 20444 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:35:40.088220 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260810 23:35:40.091589 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:35:40.092285 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:35:40.092548 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:35:40.093127 20444 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:35:40.103111 20431 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:40.103669 20431 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:35:40.105548 20431 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:35:40.107894 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:35:40.108347 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:35:40.108669 17237 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:35:40.109347 20431 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:35:44.424151 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:35:44.424652 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:35:45.174376 18140 debug-util.cc:398] Leaking SignalData structure 0x7b08000bf960 after lost signal to thread 17993
W20260810 23:35:45.175132 18140 debug-util.cc:398] Leaking SignalData structure 0x7b080008c2a0 after lost signal to thread 18143
W20260810 23:35:47.131170 17400 debug-util.cc:398] Leaking SignalData structure 0x7b08000af3a0 after lost signal to thread 17207
W20260810 23:35:47.133375 17400 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3a60 after lost signal to thread 17405
W20260810 23:35:47.859449 17189 debug-util.cc:398] Leaking SignalData structure 0x7b08000dcd60 after lost signal to thread 16999
W20260810 23:35:47.861166 17189 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4000 after lost signal to thread 17192
I20260810 23:35:48.388810 20611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:35:48.389523 20611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:35:48.391711 20611 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:35:48.395061 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:35:48.399008 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:35:48.400224 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:35:48.400660 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:35:48.401589 20611 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 23:35:48.533258 16979 heartbeater.cc:644] Failed to heartbeat to 127.11.226.189:34939 (215 consecutive failures): Network error: Failed to ping master at 127.11.226.189:34939: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
W20260810 23:35:48.685549 17195 heartbeater.cc:644] Failed to heartbeat to 127.11.226.190:42189 (247 consecutive failures): Network error: Failed to ping master at 127.11.226.190:42189: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:35:48.908116 17409 heartbeater.cc:644] Failed to heartbeat to 127.11.226.189:34939 (216 consecutive failures): Network error: Failed to ping master at 127.11.226.189:34939: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:35:49.427989 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:35:49.428586 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:35:54.432605 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:35:54.433125 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:35:55.316617 20757 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:55.317329 20757 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:35:55.319829 20757 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:35:55.322147 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:35:55.322750 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:35:55.323051 17237 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:35:55.323624 20757 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:35:59.273306 18062 maintenance_manager.cc:419] P 0bfd786d28e24be390cdbbb4007290a7: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033367
I20260810 23:35:59.381210 18059 maintenance_manager.cc:643] P 0bfd786d28e24be390cdbbb4007290a7: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.107s user 0.048s sys 0.018s Metrics: {"bytes_written":13439,"cfile_init":1,"drs_written":1,"lbm_read_time_us":264,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2560,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":25,"thread_start_us":581,"threads_started":1,"wal-append.queue_time_us":999}
I20260810 23:35:59.384162 18062 maintenance_manager.cc:419] P 0bfd786d28e24be390cdbbb4007290a7: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1362 bytes on disk
I20260810 23:35:59.386379 18059 maintenance_manager.cc:643] P 0bfd786d28e24be390cdbbb4007290a7: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":281,"lbm_reads_lt_1ms":4}
I20260810 23:35:59.437127 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:35:59.437633 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:35:59.529615 20863 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:59.530395 20863 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:35:59.532737 20863 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:35:59.534991 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:35:59.535569 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:35:59.535954 17237 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:35:59.536713 20863 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:36:04.061479 20960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:36:04.062235 20960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:36:04.064802 20960 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:36:04.067652 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:36:04.071722 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:36:04.072263 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:36:04.072553 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:36:04.073196 20960 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:36:04.440938 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:36:04.441449 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:36:06.750998 21022 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:06.752000 21022 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:36:06.754870 21022 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:36:06.757401 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:36:06.758122 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:36:06.758490 17237 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:36:06.759330 21022 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:36:09.445683 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:36:09.446203 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:36:12.842629 21156 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:12.843271 21156 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:36:12.846088 21156 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:36:12.848455 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:36:12.848902 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:36:12.849229 17237 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:36:12.849830 21156 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:36:14.449867 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:36:14.450374 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:36:18.436252 21344 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:18.437059 21344 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:36:18.439563 21344 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:36:18.441808 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:36:18.442492 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:36:18.442803 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:36:18.443588 21344 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:36:19.455467 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:36:19.456101 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:36:23.354614 21488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:36:23.355248 21488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:36:23.357756 21488 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:36:23.360540 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:36:23.363821 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:36:23.364522 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:36:23.364799 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:36:23.365422 21488 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:36:24.459913 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:36:24.460525 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:36:26.225265 21569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:36:26.225864 21569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:36:26.237514 21569 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:36:26.242178 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:36:26.242846 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:36:26.243180 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:36:26.243995 21569 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 23:36:28.565237 16972 debug-util.cc:398] Leaking SignalData structure 0x7b08000c1140 after lost signal to thread 16737
W20260810 23:36:28.566438 16972 debug-util.cc:398] Leaking SignalData structure 0x7b08000c02e0 after lost signal to thread 16976
I20260810 23:36:29.465718 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:36:29.466271 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:36:30.245016 21677 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:30.245826 21677 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:36:30.249451 21677 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:36:30.252007 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:36:30.252928 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:36:30.253366 17237 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:36:30.255146 21677 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:36:33.791775 21763 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:33.792384 21763 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:36:33.794662 21763 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:36:33.796706 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:36:33.797236 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:36:33.797580 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:36:33.798377 21763 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:36:34.469774 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:36:34.470285 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:36:35.283026 21821 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:35.283737 21821 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:36:35.286245 21821 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:36:35.287839 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:36:35.288362 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:36:35.288717 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:36:35.289364 21821 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:36:36.281141 21857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:36:36.281934 21857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:36:36.284667 21857 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:36:36.288007 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260810 23:36:36.292280 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:36:36.293087 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:36:36.293445 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:36:36.294353 21857 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:36:39.475706 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:36:39.476469 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:36:39.781958 21973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:36:39.782665 21973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:36:39.784739 21973 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:36:39.790871 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:36:39.791499 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:36:39.791811 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:36:39.792537 21973 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:36:43.995213 22081 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:43.995972 22081 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:36:43.998225 22081 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:36:44.000706 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:36:44.001302 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:36:44.001577 17237 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:36:44.002225 22081 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:36:44.480401 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:36:44.480856 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:36:45.316768 22121 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:45.317281 22121 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:36:45.319604 22121 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:36:45.321450 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:36:45.322016 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:36:45.322360 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:36:45.323069 22121 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:36:47.073233 22175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:36:47.073834 22175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:36:47.075978 22175 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:36:47.078568 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260810 23:36:47.081595 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:36:47.082118 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:36:47.082407 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:36:47.083003 22175 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 23:36:48.551039 16982 heartbeater.cc:644] Failed to heartbeat to 127.11.226.190:42189 (365 consecutive failures): Network error: Failed to ping master at 127.11.226.190:42189: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:36:48.725139 17195 heartbeater.cc:644] Failed to heartbeat to 127.11.226.190:42189 (366 consecutive failures): Network error: Failed to ping master at 127.11.226.190:42189: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:36:48.922034 17410 heartbeater.cc:644] Failed to heartbeat to 127.11.226.190:42189 (366 consecutive failures): Network error: Failed to ping master at 127.11.226.190:42189: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
I20260810 23:36:49.485152 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:36:49.485702 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:36:54.489795 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:36:54.490302 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:36:55.755659 16972 debug-util.cc:398] Leaking SignalData structure 0x7b08000e5220 after lost signal to thread 16737
W20260810 23:36:55.756933 16972 debug-util.cc:398] Leaking SignalData structure 0x7b08000c9a60 after lost signal to thread 16976
I20260810 23:36:57.818281 22498 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:57.819051 22498 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:36:57.821381 22498 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:36:57.823798 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:36:57.824517 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:36:57.825003 17237 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:36:57.825783 22498 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:36:59.494515 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:36:59.495060 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:37:02.370311 17189 debug-util.cc:398] Leaking SignalData structure 0x7b08000df100 after lost signal to thread 16999
W20260810 23:37:02.371472 17189 debug-util.cc:398] Leaking SignalData structure 0x7b0800042ce0 after lost signal to thread 17192
I20260810 23:37:02.977878 22631 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:02.978346 22631 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:37:02.980749 22631 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:37:02.982595 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:37:02.983067 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:37:02.983417 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:37:02.984105 22631 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:37:03.542151 22647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:37:03.542809 22647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:37:03.545229 22647 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:37:03.548148 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:37:03.551568 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:37:03.552309 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:37:03.552580 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:37:03.553242 22647 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 23:37:03.887111 18140 debug-util.cc:398] Leaking SignalData structure 0x7b08000bed80 after lost signal to thread 17993
W20260810 23:37:03.887975 18140 debug-util.cc:398] Leaking SignalData structure 0x7b0800051820 after lost signal to thread 18143
I20260810 23:37:04.521651 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:37:04.522079 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:37:06.963136 22765 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:06.964185 22765 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:37:06.967264 22765 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:37:06.969969 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:37:06.970708 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:37:06.971061 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:37:06.972235 22765 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:37:09.526368 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:37:09.526937 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:37:11.191063 17189 debug-util.cc:398] Leaking SignalData structure 0x7b0800043ea0 after lost signal to thread 16999
W20260810 23:37:11.192431 17189 debug-util.cc:398] Leaking SignalData structure 0x7b0800042340 after lost signal to thread 17192
I20260810 23:37:11.486928 22909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:37:11.487583 22909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:37:11.489573 22909 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:37:11.492738 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:37:11.496167 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:37:11.496840 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:37:11.497134 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:37:11.497821 22909 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:37:13.529709 22987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:37:13.530295 22987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:37:13.532408 22987 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:37:13.538275 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:37:13.538944 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:37:13.539227 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:37:13.539855 22987 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:37:14.530576 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:37:14.531075 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:37:16.784957 23083 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:16.785961 23083 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:37:16.791031 23083 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:37:16.793849 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:37:16.794651 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:37:16.795037 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:37:16.795835 23083 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:37:19.535363 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:37:19.535929 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:37:24.540230 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:37:24.540876 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:37:24.943198 17400 debug-util.cc:398] Leaking SignalData structure 0x7b0800085120 after lost signal to thread 17207
W20260810 23:37:24.944574 17400 debug-util.cc:398] Leaking SignalData structure 0x7b08000de7e0 after lost signal to thread 17405
I20260810 23:37:25.658540 23340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:37:25.659196 23340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:37:25.661032 23340 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:37:25.664378 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260810 23:37:25.668071 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:37:25.668804 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:37:25.669230 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:37:25.670007 23340 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:37:25.946004 23319 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:25.946600 23319 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:37:25.949363 23319 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:37:25.951496 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:37:25.951956 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:37:25.952365 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:37:25.953068 23319 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:37:29.545600 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:37:29.546141 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:37:34.263307 23568 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:34.264138 23568 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:37:34.266748 23568 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:37:34.269119 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:37:34.269703 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:37:34.270008 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:37:34.270694 23568 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:37:34.549593 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:37:34.550099 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:37:34.836490 23621 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:34.837023 23621 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:37:34.838842 23621 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:37:34.840610 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:37:34.841245 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:37:34.841544 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:37:34.842073 23621 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:37:39.555145 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:37:39.555671 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:37:42.139693 23820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:37:42.140486 23820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:37:42.143455 23820 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:37:42.146427 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:37:42.149765 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:37:42.150418 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:37:42.150656 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:37:42.151235 23820 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:37:44.559548 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:37:44.560164 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:37:48.262138 16972 debug-util.cc:398] Leaking SignalData structure 0x7b08000e2020 after lost signal to thread 16737
W20260810 23:37:48.264065 16972 debug-util.cc:398] Leaking SignalData structure 0x7b08000de480 after lost signal to thread 16976
W20260810 23:37:48.552588 16982 heartbeater.cc:644] Failed to heartbeat to 127.11.226.190:42189 (484 consecutive failures): Network error: Failed to ping master at 127.11.226.190:42189: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:37:48.770367 17195 heartbeater.cc:644] Failed to heartbeat to 127.11.226.190:42189 (485 consecutive failures): Network error: Failed to ping master at 127.11.226.190:42189: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:37:48.969879 17409 heartbeater.cc:644] Failed to heartbeat to 127.11.226.189:34939 (454 consecutive failures): Network error: Failed to ping master at 127.11.226.189:34939: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:37:49.563946 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:37:49.564508 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:37:50.026355 24035 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:50.027519 24035 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:37:50.030473 24035 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:37:50.033315 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:37:50.033967 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:37:50.034272 17237 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:37:50.034906 24035 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:37:51.417865 24080 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:51.418489 24080 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:37:51.420596 24080 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:37:51.422667 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:37:51.423111 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:37:51.423478 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:37:51.424264 24080 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:37:53.044070 24128 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:53.044579 24128 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:37:53.046558 24128 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:37:53.048801 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:37:53.049314 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:37:53.049707 17237 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:37:53.050417 24128 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:37:54.567593 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:37:54.568092 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:37:58.622634 24266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:37:58.623301 24266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:37:58.625926 24266 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:37:58.629231 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:37:58.632627 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:37:58.633364 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:37:58.633626 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:37:58.634295 24266 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:37:59.572649 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:37:59.573315 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:38:03.396659 24408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:38:03.397315 24408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:38:03.399608 24408 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:38:03.405048 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:38:03.405824 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:38:03.406121 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:38:03.406792 24408 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:38:03.507701 24402 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:03.508523 24402 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:38:03.510684 24402 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:38:03.513427 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:38:03.514178 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:38:03.514541 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:38:03.515331 24402 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:38:04.576721 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:38:04.577184 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:38:06.525338 24484 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:06.526036 24484 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:38:06.528411 24484 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:38:06.530628 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:38:06.531103 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:38:06.531435 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:38:06.532073 24484 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:38:07.840605 24542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:38:07.844087 24542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:38:07.846670 24542 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:38:07.849968 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260810 23:38:07.853814 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:38:07.854532 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:38:07.854777 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:38:07.855424 24542 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:38:09.558413 24585 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:09.559278 24585 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:38:09.561558 24585 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:38:09.563884 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:38:09.564580 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:38:09.564846 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:38:09.565493 24585 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:38:09.581146 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:38:09.581637 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:38:14.585378 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:38:14.585964 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:38:19.590077 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:38:19.590627 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:38:20.954119 24894 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:20.954927 24894 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:38:20.957250 24894 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:38:20.959523 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:38:20.960103 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:38:20.960424 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:38:20.961169 24894 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:38:24.594008 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:38:24.594483 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:38:26.976913 25068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:38:26.977586 25068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:38:26.980304 25068 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:38:26.991081 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:38:26.995980 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:38:26.996907 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:38:26.997220 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:38:26.997879 25068 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:38:29.546535 25128 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:29.547510 25128 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:38:29.550800 25128 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:38:29.553161 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:38:29.553905 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:38:29.554203 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:38:29.554963 25128 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:38:29.598912 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:38:29.599448 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:38:32.420364 25210 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:32.420985 25210 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:38:32.422892 25210 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:38:32.424871 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:38:32.425351 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:38:32.425694 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:38:32.426347 25210 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:38:34.603091 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:38:34.603672 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:38:39.051405 25392 raft_consensus.cc:482] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:39.052222 25392 raft_consensus.cc:504] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } }
I20260810 23:38:39.054728 25392 leader_election.cc:288] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:38:39.056957 17237 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
W20260810 23:38:39.057543 17234 leader_election.cc:341] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 14a2f1094d9d43aa8250f196faf19c69
I20260810 23:38:39.057826 17234 leader_election.cc:302] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:38:39.058559 25392 raft_consensus.cc:2726] T 14a2f1094d9d43aa8250f196faf19c69 P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:38:39.616178 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:38:39.616963 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:38:40.654673 17400 debug-util.cc:398] Leaking SignalData structure 0x7b08000b4760 after lost signal to thread 17207
W20260810 23:38:40.655925 17400 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3240 after lost signal to thread 17405
I20260810 23:38:41.407531 25471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:38:41.408215 25471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:38:41.410349 25471 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:38:41.413100 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:38:41.416235 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:38:41.416888 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:38:41.417141 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:38:41.417964 25471 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260810 23:38:41.664767 25457 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:41.665352 25457 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:38:41.667376 25457 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:38:41.669512 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:38:41.669967 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:38:41.670357 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:38:41.671137 25457 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:38:44.620378 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:38:44.620922 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:38:44.633323 17189 debug-util.cc:398] Leaking SignalData structure 0x7b08000e6ac0 after lost signal to thread 16999
W20260810 23:38:44.634599 17189 debug-util.cc:398] Leaking SignalData structure 0x7b08000dfa20 after lost signal to thread 17192
W20260810 23:38:48.590070 16982 heartbeater.cc:644] Failed to heartbeat to 127.11.226.190:42189 (603 consecutive failures): Network error: Failed to ping master at 127.11.226.190:42189: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:38:48.779938 17194 heartbeater.cc:644] Failed to heartbeat to 127.11.226.189:34939 (572 consecutive failures): Network error: Failed to ping master at 127.11.226.189:34939: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
W20260810 23:38:49.004772 18140 debug-util.cc:398] Leaking SignalData structure 0x7b08000b7660 after lost signal to thread 17993
W20260810 23:38:49.005636 18140 debug-util.cc:398] Leaking SignalData structure 0x7b08000b7f20 after lost signal to thread 18143
W20260810 23:38:49.012145 17409 heartbeater.cc:644] Failed to heartbeat to 127.11.226.189:34939 (573 consecutive failures): Network error: Failed to ping master at 127.11.226.189:34939: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:38:49.669073 17981 master_runner.cc:186] Error getting master registration for 127.11.226.190:42189: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111),
I20260810 23:38:49.669639 17981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:38:49.688098 25697 raft_consensus.cc:482] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:38:49.688851 25697 raft_consensus.cc:504] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b587fa042b42e4b66bb113245c0e36" member_type: VOTER last_known_addr { host: "127.11.226.129" port: 39773 } } peers { permanent_uuid: "80a651570d6e459d98bcf97054e07921" member_type: VOTER last_known_addr { host: "127.11.226.130" port: 42517 } } peers { permanent_uuid: "2466b33eeb034688bcf670ff612d452a" member_type: VOTER last_known_addr { host: "127.11.226.131" port: 39949 } }
I20260810 23:38:49.691470 25697 leader_election.cc:288] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773), 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517)
W20260810 23:38:49.693485 17237 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b587fa042b42e4b66bb113245c0e36 (127.11.226.129:39773): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
W20260810 23:38:49.694157 17234 leader_election.cc:341] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a651570d6e459d98bcf97054e07921 (127.11.226.130:42517): Not found: Tablet not found: 22d4f61906a14dc58a9c329b97c79d9a
I20260810 23:38:49.694494 17234 leader_election.cc:302] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2466b33eeb034688bcf670ff612d452a; no voters: 80a651570d6e459d98bcf97054e07921, c3b587fa042b42e4b66bb113245c0e36
I20260810 23:38:49.695139 25697 raft_consensus.cc:2726] T 22d4f61906a14dc58a9c329b97c79d9a P 2466b33eeb034688bcf670ff612d452a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:38:50.340945 25711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0596cdb0a514cc08d7df0ceea528339)
I20260810 23:38:50.341574 25711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61534c7c271c4e3393ff5b46a3498f9b" member_type: VOTER last_known_addr { host: "127.11.226.190" port: 42189 } } peers { permanent_uuid: "d0596cdb0a514cc08d7df0ceea528339" member_type: VOTER last_known_addr { host: "127.11.226.189" port: 34939 } } peers { permanent_uuid: "0bfd786d28e24be390cdbbb4007290a7" member_type: VOTER last_known_addr { host: "127.11.226.188" port: 46109 } }
I20260810 23:38:50.344399 25711 leader_election.cc:288] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189), d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939)
W20260810 23:38:50.347150 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260810 23:38:50.350584 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 61534c7c271c4e3393ff5b46a3498f9b (127.11.226.190:42189): Network error: Client connection negotiation failed: client connection to 127.11.226.190:42189: connect: Connection refused (error 111)
W20260810 23:38:50.351250 18056 leader_election.cc:334] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0596cdb0a514cc08d7df0ceea528339 (127.11.226.189:34939): Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
I20260810 23:38:50.351511 18056 leader_election.cc:302] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfd786d28e24be390cdbbb4007290a7; no voters: 61534c7c271c4e3393ff5b46a3498f9b, d0596cdb0a514cc08d7df0ceea528339
I20260810 23:38:50.352150 25711 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bfd786d28e24be390cdbbb4007290a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260810 23:38:50.847820 17428 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109, num_attempts: 754) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
W20260810 23:38:50.848774 17428 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109, num_attempts: 754) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
W20260810 23:38:50.849119 17428 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109, num_attempts: 754) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
W20260810 23:38:50.849423 17428 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109, num_attempts: 754) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
F20260810 23:38:50.861406 17447 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bf8e0e4f072044999c5d464208e52921 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109, num_attempts: 754) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260810 23:38:50.863111 17444 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_585e701700f14afbb2df4af6ca5e03f5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109, num_attempts: 754) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
*** Aborted at 1786405130 (unix time) try "date -d @1786405130" if you are using GNU date ***
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800002f8a) received by PID 12170 (TID 0x7faed91fb700) from PID 12170; stack trace: ***
@ 0x2b0af0 __tsan::CallUserSignalHandler() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b31e0 rtl_sigaction() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
W20260810 23:38:50.894649 17428 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.226.189:34939)
@ 0x7faeebeae980 (unknown) at ??:0
@ 0x7faee716be87 gsignal at ??:0
@ 0x7faee716d7f1 abort at ??:0
@ 0x2b4f37 __interceptor_abort at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7faeea19fdf4 google::LogMessage::SendToLog() at ??:0
@ 0x7faeea1a09a0 google::LogMessage::Flush() at ??:0
@ 0x7faeea1a5adb google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x3572c4 kudu::MasterStressTest::CreateTableThread() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x3874ce _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3840
F20260810 23:38:50.863111 17444 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_585e701700f14afbb2df4af6ca5e03f5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109, num_attempts: 754) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)F20260810 23:38:50.895149 17448 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.226.190:42189,127.11.226.189:34939,127.11.226.188:46109, num_attempts: 754) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.226.189:34939: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x387381 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x386acb std::__1::__thread_proxy<>() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2b11b0 __tsan_thread_start_func at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7faeebea36db start_thread at ??:0
@ 0x7faee724e61f clone at ??:0